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Executive  Summary 

This  report  describes  a  28-month  Phase  II  SBIR  project  that  modeled  pilot  spatial  disorientation 
(SD)  illusions  as  part  of  a  real-time  illusion  detection  and  aiding  system.  The  models  and 
detection  algorithms  focus  on  human  vestibular  responses  to  aircraft  motions.  The  detection 
subsystem  purposely  excludes  orientation  cues  from  other  senses  (e.g.,  vision)  because 
contributions  from  those  senses  have  not  been  quantified.  Detection  relies  upon  computational 
models  of  four  illusions  that  are  flight  hazards:  leans,  graveyard  spiral  (both  are  somatogyral 
illusions),  somatogravic,  and  Coriolis. 

The  aiding  portion  of  the  product  is  part  of  a  comprehensive  approach  to  combating  the  adverse 
effects  of  SD  on  military  and  civilian  pilots.  A  comprehensive  approach  includes:  pilot 
selection  and  training  (which  has  shown  recent  successes);  mission  pre-briefings  that  emphasize 
SD  susceptibility  (e.g.,  when  flying  at  night  over  featureless  terrain);  hypothetical  intelligent 
cockpit  aids  that  help  pilots  retain  orientation  and  aid  recovery  when  they  are  disoriented;  post¬ 
mission  analyses  of  flight  data;  and,  a  method  for  feeding  back  lessons  learned  from  each 
approach  component  to  the  others.  The  components  of  focus  for  this  project  were  the  cockpit  aid 
(which  uses  detection  and  aiding)  and  the  post-flight  analyses  (where  detection  is  key). 

In  the  aiding  subsystem,  the  multisensory  countermeasures  hypothesized  cues  and  commands  in 
the  following  modalities:  visual,  auditory,  tactile,  and  olfactory.  In  the  interest  of  completeness, 
we  also  hypothesized  how  an  intelligent  (context  sensitive)  aiding  system  could  trigger  auto¬ 
recovery  (e.g.,  auto-GCAS)  and  auto-ejection  (as  a  last  resort  to  save  the  pilot’s  life,  if  the 
aircraft  is  damaged  and  unable  to  recover).  The  countermeasures  were  triggered  based  on  three 
criteria:  The  detection  portion’s  assessment  of  level  of  certainty  that  an  SD  event  is  occurring, 
the  aircraft  state,  and  the  pilot’s  multisensory  workload.  For  aircraft  state,  the  main 
considerations  are  assessments  of  unusual  attitudes  and  time  to  ground  impact.  For  pilot 
workload,  aiding  suppresses  countenneasures  in  modalities  in  which  the  pilot  workload  is  high. 
For  example,  during  radio  chatter,  audio  countermeasures  are  suppressed  in  favor  of  visual  and 
tactile  ones. 

Four  experiments  supported  the  detection  and  aiding  calculations.  Two  focused  on  creating 
vestibular  illusions  and  quantifying  pilot  perception  of  those  illusions  for  our  models.  Data  from 
the  experiments  were  used  in  the  models  and  included  the  exponential  decay  time  constant  for 
sustained  rotations,  and  the  magnitude  of  illusory  rotation  after  halting  a  sustained  rotation.  Two 
experiments  focused  on  the  aiding  portion  of  the  system  and  tested  an  innovative  visual  cue, 
called  the  SD  Icon,  as  well  as  the  efficacy  of  voice  and  tactile  commands  for  recovering  from 
unusual  attitudes.  For  three  of  the  experiments,  we  drafted  and  submitted  journal  manuscripts. 

The  vestibular  models  formed  the  foundation  of  a  tool  for  post  hoc  flight  data  analyses  from  SD 
mishaps.  We  analyzed  mishap  data  from  nine  Air  Force  mishaps  (some  fatal),  one  fatal  Navy 
mishap,  and  two  fatal  civilian  accidents.  The  analyses  were  invaluable  in  fine-tuning  the  tool 
and  our  analysis  process. 
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Other  accomplishments  include  a  patent  application,  professional  conference  presentations,  a 
usability  evaluation  of  our  software  product,  and  a  preliminary  examination  of  how  to  apply  our 
results  to  the  huge  medical  problem  of  elderly  person  imbalances  and  falls. 

Recommendations  for  future  work  are  to: 

•  Conduct  more  experiments,  specifically  to  quantify  visual  dominance,  and  to  test  our 
aiding  system  in  actual  flight; 

•  Analyze  more  mishaps  because  the  more  data  we  analyze,  the  more  we  leam  and  the 
better  are  our  models; 

•  Assist  an  accident  investigation  organization  (e.g.,  Air  Force  Safety  Center,  National 
Transportation  Safety  Board)  with  ongoing  accident  investigations  where  SD  is 
suspected; 

•  Analyze  combat  maneuvering  data  (e.g.,  from  Red  Flag  exercises)  to  find,  and  adjust  for, 
false  positives; 

•  Apply  our  detection  and  aiding  tools  to  UAV  applications;  and, 

•  Apply  our  detection  and  aiding  tools  to  other  domains  -  astronauts,  divers,  firefighters, 
and  gerontology  (since  falls  by  elderly  people  are  a  huge  medical  expense  in  the  US  and 
elsewhere). 

Readers  interested  in  further  details,  or  in  pursuing  these  recommendations  or  others  of  interest 
to  them,  should  contact  the  first  author  via  phone  (303.442.6947  xl65)  or  via  email  at 
rsmall@alionscience.com. 
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1  Introduction 

Spatial  disorientation  (SD1 2)  continues  to  be  an  enduring  problem  in  aviation.  For  example, 
during  the  1980s  and  90s,  SD  was  a  factor  in  approximately  21%  of  all  US  Air  Force  Class  A 
mishaps.  Between  October  1993  and  September  2002,  there  were  25  high-performance  fighter 
or  attack  mishaps  where  SD  was  identified  as  a  causal  or  contributing  factor.  These  mishaps 
resulted  in  19  fatalities  and  cost  the  Air  Force  over  $455  million  (Sundstrom,  2004).  In  the 
United  Kingdom,  SD  mishap  frequency  is  estimated  to  be  between  6  and  21%,  depending  on 
branch  of  service  (Holmes  et  ah,  2003).  Furthennore,  SD  accidents  are  often  fatal;  in  civilian 
aviation,  Wiegmann  et  al.  (2004)  observed  that  over  90%  of  SD  accidents  resulted  in  fatalities. 

Spatial  disorientation,  simply  referred  to  as  “vertigo”  until  becoming  the  subject  of  scientific 
research,  is  a  nonnal  response  to  neuro-physiologically  confusing  stimuli  in  the  flight 
environment.  The  human  body  developed  over  eons  to  perceive  motion  on  land  in  relation  to  the 
surface  of  the  earth.  In  flight,  however,  these  physiological  systems  can  give  the  brain  erroneous 
orientation  information,  resulting  in  SD.  Most  pilots  will  eventually  experience  SD;  some  pilots 
will  experience  it  more  acutely  than  others.  In  military  aviation,  attention  mismanagement  is  a 
major  contributing  factor.  In  civilian  flying  (general  aviation),  accidentally  straying  into 
degraded  visual  conditions  (e.g.,  flying  into  clouds)  is  the  major  contributing  factor  in  this 
domain  (Nall,  1999). 

For  this  research,  we  use  the  definition  of  SD  from  Previc  and  Ercoline,  who  quote  Benson:  “SD 
refers  to  the  pilot’s  ‘failure  to  sense  correctly  the  position,  motion  or  attitude  of  his  aircraft  or  of 
him/herself  within  the  fixed  coordinate  system  provided  by  the  surface  of  the  earth  and  the 
gravitational  vertical.’”  (Benson  quoted  in  Previc  &  Ercoline,  2004,  pg  1).  This  definition 
focuses  on  the  issue  of  “which  way  is  up?”  while  ignoring  geographical  orientation,  “where  am 
I?”  The  fonner  is  also  our  emphasis,  with  a  further  focus  on  the  vestibular  perception  of  which 
way  is  up  (pitch  and  roll  perception),  and  which  way  is  forward  along  the  desired  path  (yaw 
axis/heading  perception).  Even  though  a  human’s  vision  is  dominant  in  determining  spatial 
orientation,  we  ignore  it  within  our  models  because  there  is  no  way  that  we  know  of  (or 
presented  in  the  literature)  to  quantify  such  visual  dominance.  Quantifying  visual  dominance  is 
an  area  ripe  for  further  research,  as  suggested  in  Chapter  4. 

Our  research  is  part  of  a  comprehensive  approach  to  the  SD  problem  that  includes:  pilot 
selection  (potentially),  pilot  training  (which  has  shown  promise  over  recent  years  already), 
mission  pre-briefings  (that  emphasize  SD  susceptibility;  e.g.,  when  flying  at  night  over 
featureless  desert  or  the  ocean),  intelligent  cockpit  aiding  systems  (such  as  our  SOAS),  and 
rigorous  post-event  analyses  (such  as  can  be  done  with  SDAT).  The  results  of  rigorous  analyses 
feed  back  into  the  other  means  of  combating  SD  to  ensure  a  constantly-improving 
comprehensive  approach. 


1  All  acronyms  are  defined  in  the  Glossary. 

2  Class  A  mishaps  are  those  where  there  is  loss  of  life,  injury  resulting  in  permanent  total  disability,  destruction  of  an 
aircraft,  and/or  property  damage/loss  exceeding  $1  million  (Air  Force,  2001). 
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2  Overview 

The  research  effort  reported  herein  was  a  typical  SBIR:  a  6-month  Phase  I  proof-of-concept 
project  followed  by  a  2-year  development  project.  Because  Alion/MA&D’s  niche  in  the  human 
factors  R&D  market  is  human  performance  modeling,  that  is  how  we  approached  the  problem. 

In  Phase  I  we  modeled  two  common  SD  illusions:  Leans  and  Coriolis.  We  selected  those 
illusions  because  Leans  is  the  most  common  SD  illusion,  and  is  a  slow-onset  insidious  hazard. 
Coriolis,  on  the  other  hand,  is  a  quick  onset  illusion  and  often  fatal.  Thus,  we  strove  to  capture 
opposite  ends  of  the  SD  illusion  spectrum,  in  a  sense.  (In  actuality,  SD  illusions  are  multi¬ 
dimensional,  so  timing  and  severity  are  just  two  ways  to  potentially  categorize  them.) 

Proving  the  value  of  our  approach  in  Phase  I  led  directly  into  Phase  II,  where  we  more  fully 
developed  our  concepts,  models,  and  tools.  It  is  important  to  note  that  one  commonly  known 
benefit  to  computer  modeling  of  anything  is  that  the  modeling  team  leams  a  great  deal  about  the 
phenomenon  being  modeled  and  the  problem  being  addressed  by  the  modeling  effort.  So,  even 
if  we  had  been  unsuccessful  with  our  approach,  we  would  have  contributed  to  the  research  base. 

The  emphasis  of  this  report  is  on  Phase  II.  Readers  interested  in  the  full  evolution  of  our  thought 
process  should  review  our  Phase  I  final  report  (Small  et  ah,  2004). 

As  we  evolved  our  approach  and  discussed  ideas  with  colleagues  and  collaborators  (see  Ac¬ 
knowledgments),  we  began  to  see  a  theme:  Detection  and  Aiding  (Figure  1).  Readers  will 
notice  this  theme  throughout  this  report.  Detection  means  all  of  the  work  done  to  understand  the 
nature  of  SD  illusions  and  to  detect  them  within  flight  data,  simulator  data,  and  even  experiential 
anecdotes.  Aiding  refers  to  our  approach  to  minimizing  SD  illusion  severity  by  applying 
countermeasures  and  otherwise  aiding  pilots  -  helping  them  to  recover  from  illusions.  Figure  1 
also  highlights  the  project’s  two  products  -  spatial  disorientation  analysis  tool  (SDAT),  and 
spatial  orientation  aiding  system  (SOAS)  -  which  are  actually  combined  into  a  single  software 
system  as  a  deliverable  product. 


SD  Detection 

+ 

Pilot  Aiding 

>  Vestibular-based  models 
of  attitude  perception 

>  Illusion  sequence  models 

>  SD  certainty  prediction 

>  Pilot  workload  model 

>  Aircraft  risk  model 

>  Layered  pilot  support 
countermeasures 

SDAT 

Spatial  Disorientation 
Analysis  Tool 

>  Post  event  flight  data 
analysis 

>  User  modified  model 
variables 

>  Countermeasure  sequence 
demonstration 

>  User  modified  initiation 
variables 

SOAS 

Spatial  Orientation 
Aiding  System 

>  Real-time  integrated 

cockpit  monitoring  system 

>  Real-time  integrated  cockpit 
countermeasure  initiation 
system 

Figure  1.  Overview  of  project’s  technical  approach. 
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We  would  have  considered  ourselves  successful  if  we  had  only  been  able  to  reliably  detect  SD 
illusions.  But,  as  human  factors  professionals  whose  goal  is  always  to  help  users  (in  this  case, 
pilots),  we  wanted  to  do  more.  Detecting  SD  is  necessary,  but  not  sufficient,  to  saving  lives.  To 
illustrate  this  point,  our  detection  system  could  fairly  reliably  inform  pilots  that  they  have  SD. 
But,  merely  informing  them  of  the  hazard  does  only  half  the  job  -  and  the  less  important  part  at 
that  (in  our  view).  Because  SD  is,  by  definition,  disorienting,  it  is  vital  to  help  the  pilots 
minimize  the  adverse  consequences  of  the  SD  event.  Therefore,  we  also  explored  ways  to  help 
pilots  recover  from  SD  when  it  happens  (not  if  it  happens).  And,  ultimately,  that  was  our 
customer’s  primary  goal:  to  devise  a  set  of  integrated  multisensory  methods  to  help  pilots 
maintain  spatial  orientation  (SO).  If  pilots  maintain  SO  all  the  time,  then  they  would  never 
succumb  to  SD.  While  this  is  a  noble  goal,  it  is  unrealistic.  The  vast  majority  of  pilots  suffer 
from  SD  at  some  point  in  their  flying  careers  (Previc  &  Ercoline,  2004);  it  is  nearly  unavoidable. 
As  part  of  a  comprehensive  approach,  it  is  simply  insufficient  to  combat  SD  from  a  pure 
prevention  standpoint  -  plus  such  an  approach  is  doomed  to  failure,  as  already  emphasized. 
Therefore,  we  needed  to  pay  attention  to  detection  and  aiding. 

Some  have  argued  that,  by  removing  human  pilots  from  aircraft  and  using  UAVs  for  missions, 
we  will  eliminate  the  SD  problem.  This  is  wishful  thinking,  as  the  accident  rate  of  some  UAVs 
attest.  Some  UAVs  experience  a  near  50%  loss  rate  due,  in  large  part,  to  visual  display  illusions 
during  the  landing  phase  of  the  mission.  So,  removing  human  pilots  from  aircraft  does  not  solve 
the  SD  problem,  it  merely  shifts  the  problem  to  those  SD  illusions  from  which  UAV  operators 
suffer. 

SD  detection  relies  upon  models  that  reflect  our  understanding  of  specific  SD  illusions,  as  well 
as  the  ways  pilots  get  into  trouble  from  a  spatial  orientation  perspective.  There  is  a  vast  literature 
base  that  helped  us  understand  SD.  However,  it  has  some  gaps,  which  naturally  prompted  us  to 
conduct  experiments  to  fill  some  of  the  gaps.  Our  experimental  results  fed  directly  into  our 
modeling  effort,  and  also  served  to  expand  the  research  base.  Appendix  A  details  all  of  our 
experiments  in  journal  papers  that  we  have  submitted  for  publication. 

Aiding  pilots  who  suffer  from  SD  has  also  received  considerable  research  attention.  The  typical 
methods  are  to  use  enhanced  visual  displays,  novel  auditory  displays,  and  even  haptic  or  tactile 
displays.  Again,  we  found  gaps  in  the  research  base,  and  so  designed  and  conducted  experiments 
to  test  our  thoughts  about  the  best  way  to  help  pilots.  Appendix  A  also  details  these  experiments 
and  results. 

The  culmination  of  our  approach  to  improving  detection  and  aiding  is  a  software  tool  that  serves 
both  purposes.  We  call  the  detection  and  analysis  portion  SDAT  -  spatial  disorientation  analysis 
tool.  It  is  explained  in  Chapter  3,  and  further  detailed  in  its  User  Guide  (Alion,  2006).  The 
emphasis  of  SDAT  is  on  post  hoc  data  set  analyses. 

The  aiding  portion  is  SOAS  -  spatial  orientation  aiding  system.  SOAS  is  intended  to  be  a  real¬ 
time  in  situ  cockpit  system  that  helps  pilots  recover  from  SD  and  to  avoid  its  bad  consequences 
(e.g.,  crashes).  Importantly,  SDAT  and  SOAS  use  the  same  code  base  which  runs  faster  than 
real  time,  and  we  used  SDAT  extensively  in  Phase  II  to  analyze  actual  SD  mishap  flight  data  sets 
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(Section  3.5).  Due  to  schedule  and  budget  limitations,  we  were  unable  to  test  SOAS  in  flight 
with  real  pilots.  That  is  our  greatest  disappointment.  However,  adequately  instrumented  aircraft 
are  uncommon,  therefore  difficult  to  schedule,  and  they  are  expensive.  Near  the  end  of  Phase  II, 
we  found  one  at  the  University  of  Iowa,  and  hope  to  use  it  in  Phase  III,  if  we  are  fortunate 
enough  to  attract  sufficient  additional  funding. 

The  last  chapter  of  this  report  (Chapter  4)  recommends  many  steps  to  consider  in  Phase  III  or  in 
any  similar  follow-on  research.  While  we  are  very  proud  of  our  results,  we  recognize  there  is 
still  much  to  do  in  both  areas  -  detection  and  aiding.  Aiding,  especially,  needs  attention,  as 
certifying  cockpit  systems  is  an  intensive  multi-year  effort. 


2. 1  Technical  Approach 

The  project  team’s  technical  approach  was  to  understand  and  model  SD,  and  to  devise  multisen- 
sory  countermeasures  for  helping  pilots  who,  despite  the  best  intentions  and  pre -mission 
preparations,  experience  SD;  it  is  a  nearly  unavoidable  problem.  Therefore,  we  had  to 
understand  the  root  causes  of  SD,  scope  the  problem  to  fit  within  a  Phase  II  SBIR  project,  model 
illusions,  develop  a  tool  (SDAT)  to  try  our  detection  algorithms,  test  it  and  refine  it,  and  then 
focus  on  the  aiding  component  by  developing  a  countenneasure  or  compensatory  piece  (SOAS) 
once  we  felt  we  had  done  an  adequate  job  with  the  detection  challenge.  Even  though  this 
description  implies  a  linear  progression  from  start  to  finish,  in  actuality  it  was  an  iterative 
process  with  multiple  feedback  loops,  some  dead-ends,  and  the  usual  trial-and-error  efforts  of  a 
typical  R&D  project.  That  is,  we  tried  some  detection  and  aiding  ideas  in  parallel,  saw  what 
worked  and  what  did  not,  and  then  iterated  until  we  ran  out  of  time  and  budget. 

We  learned  fairly  early  in  the  research  that  we  could  not  hope  to  detect  every  known  SD  illusion. 
For  example,  there  are  many  visual  illusions  that  simply  do  not  lend  themselves  to  detection  via 
a  flight  data  processing  approach,  no  matter  how  many  novel  sensors  we  cared  to  imagine. 
Therefore,  we  scoped  the  effort  to  focus  on  vestibular  illusions,  and,  more  specifically,  on  those 
that  could  be  detected  via  flight  data  processing  -  data  that  are  available  on  today’s  aircraft  or  via 
today’s  national  airspace  system  (NAS)  technologies  (e.g.,  enroute  and  tenninal  radars),  rather 
than  some  hypothetical  future  aircraft  equipped  with  pilot  physiological  monitoring,  for  example. 

Section  2.3  details  the  physiology  that  we  learned  primarily  during  Phase  I;  we  continued  to  add 
to  our  knowledge  during  Phase  II,  especially  with  the  publication  of  the  Previc  and  Ercoline 
book  (2004).  The  physiological  knowledge  directly  supports  our  detection  (modeling)  efforts. 
We  detennined  that  there  are  four  ways  that  a  pilot’s  vestibular  system  can  be  “tricked”  by 
typical  aircraft  motions.  These  four  ways  are: 

1 .  Sub-threshold  rotations  that  the  vestibular  system  does  not  register,  even  though  the  aircraft 
moves.  These  sub-threshold  rotations  typically  occur  around  the  roll  axis  of  the  aircraft. 

2.  Sustained  rotations  that  yield  washout,  so  that  the  vestibular  system  stops  sensing  the 
rotation,  even  though  the  aircraft  is  still  rotating.  Washout  typically  occurs  in  the  yaw 
(heading)  plane  of  aircraft  motion. 

3.  After  washout,  returning  to  straight  flight  (as  occurs  when  rolling  out  from  a  sustained  turn) 
yields  illusory  rotation.  That  is,  the  vestibular  system  perceives  a  turn  in  the  opposite 
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direction  from  the  original  turn.  Eventually,  there  is  washout  of  the  illusory  rotation;  this 
illusory  washout  occurs  at  the  same  rate  as  washout  from  an  actual  turn. 

4.  Changes  in  airspeed  yield  perceptions  of  pitch  changes.  Decelerations  feel  like  pitching 
down;  accelerations  feel  like  pitching  up  from  the  actual  pitch  angle. 

Our  models  and  vestibular  attitude  calculations  constantly  compare  the  actual  aircraft  motions 
(and  resulting  attitude)  with  what  the  pilot’s  vestibular  system  perceives.  Differences  between 
actual  and  vestibular-perceived  motions  and  attitudes  are  the  basis  for  detecting  SD.  A  more 
detailed  explanation  of  the  human  vestibular  system  is  in  Section  2.3.  More  details  about  our 
modeling  and  vestibular  attitude  calculator  (VAC)  methods  are  in  Section  3.1. 


2.2  Phase  I  Highlights 

In  Phase  I  of  this  SBIR,  we  accomplished  the  following  major  objectives: 

•  A  literature  search,  including  Russian  literature,  which  formed  the  foundation  of  our 
quantitative,  model-based  approach.  An  annotated  bibliography  is  Chapter  IX  of  the 
Phase  I  final  report. 

•  Designed  and  developed  detection  models  for  the  Leans  and  Coriolis  illusions.  The 
Leans  model  worked  with  the  actual  L-16  data  that  we  received  from  the  Air  Lorce  Safety 
Center  (ALSC)  within  the  last  weeks  of  the  Phase  I  technical  work.  We  were  especially 
delighted  that  our  SD  Aiding  system  (the  name  was  changed  to  SOAS  in  Phase  II)  was 
robust  enough  to  detect  instances  of  SD  within  this  data  set  in  a  sensible,  realistic 
manner. 

•  Enhanced  Wickens’  (1984)  Multiple  Resource  Theory  to  account  for  the  other  senses 
central  to  this  multisensory  effort.  We  called  the  resulting  “theory”  SAVVOY  in  honor 
of  Dr.  Wickens’  location  in  Savoy,  Illinois  (at  the  University  of  Illinois’  Aviation 
Research  Laboratory).  The  newly  included  senses  were  somatic,  vestibular,  and 
olfactory.  A  model  of  pilot  workload,  using  hypothetical  (but  realistic)  SAVVOY  scores, 
influenced  the  SD  countermeasures  applied  by  our  SD  Aiding  System.  The  acronym 
SAVVOY  represents  somatic,  auditory,  visual,  vestibular,  olfactory,  and  psychomotor. 
Pilot  workload  is  an  important  factor  in  determining  which  aiding  countermeasures  to 
apply,  and  when.  Lor  example,  when  there  is  radio  chatter,  we  suppress  audio 
countenneasures  in  favor  of  visual  and  tactile  ones. 

•  Invented  an  SD  Icon  to  be  an  intuitive  representation  of  the  aircraft  pitch  and  roll.  While 
we  originally  hypothesized  the  SD  Icon  as  a  status  display,  we  detennined  that  it  might 
be  more  valuable  as  a  command  display.  Phase  II  aiding  experiments  tested  the  value  of 
the  Icon. 

•  Most  importantly,  we  delivered  a  working  SD  Aiding  prototype  that  served  as  an  excel¬ 
lent  foundation  for  Phase  II  development  and  testing.  The  software  grew  considerably  in 
Phase  II.  Both  detection  and  aiding  were  improved. 


Ligure  2  summarizes  and  highlights  Phase  I.  It  emphasizes  the  aiding  part  of  the  SD  challenge 
because  that  is  what  made  for  a  compelling  demonstration  to  help  us  win  Phase  II.  It  proposed 
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layered  or  sequential  multisensory  countermeasures,  including  visual,  auditory,  tactile,  and 
olfactory  cues,  as  well  as  combinations  of  these  to  help  pilots  notice  their  unusual  attitude  (due  to 
SD)  and  prompted  their  recovery  actions.  As  a  proof-of-concept,  we  did  not  have  to  make  sure 
that  everything  worked  in  our  conceptual  prototype,  but  it  did.  We  accounted  for  the  pilot’s 
multisensory  workload  and  the  aircraft  situation.  For  example,  if  the  aircraft  was  about  to  crash, 
the  countermeasures  were  more  intrusive  (e.g.,  combined  visual,  audio,  and  tactile  recovery 
commands).  If  the  SD  event  happened  at  high  altitude,  such  that  its  consequences  were  less 
severe,  then  the  countenneasures  were  less  intrusive  (e.g.,  visual  cue  only).  In  either  case,  the 
assessment  of  the  confidence  level  that  the  modeled  SD  illusion  was  actually  occurring  also 
contributed  to  the  prototype’s  triggering  of  countermeasures. 


Our  innovative 
“SD  icon” 
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Figure  2.  Phase  1  aiding  concept  screen  shot. 


2.3  SD  Physiology  Background 

In  addition  to  initiating  detection  and  aiding  concepts  in  Phase  I  and  carrying  them  forward  into 
Phase  II,  another  important  contribution  from  Phase  I  into  Phase  II  is  the  physiology  of  SD  - 
with  emphasis  on  the  human  vestibular  system.  The  goal  of  modeling  SD  illusions  in  real  time  is 
to  compare  actual  aircraft  attitude  to  our  inference  of  the  pilot’s  perception  of  aircraft  attitude.  It 
is  the  latter  that  our  basic  models  capture  via  our  Vestibular  Attitude  Calculator  (VAC).  The 
relevant  physiology  of  SD  follows  so  that  readers  do  not  have  to  look  elsewhere  for  this  vital 
background  infonnation. 

The  eyes,  the  most  important  source  of  motion  information  to  the  human  brain,  send  information 
to  the  pilot’s  brain  about  the  aircraft’s  position,  velocity,  and  attitude  relative  to  the  ground.  The 
overwhelming  reliance  on  visual  cues  over  those  from  other  senses,  when  there  are  sufficient 


visual  cues,  is  called  visual  dominance.  Visual  cues  are  ideal  for  spatial  orientation  on  clear  days 
in  VFR  conditions  with  a  well-defined  horizon.  But,  when  visibility  is  poor  (e.g.,  during  night 
flying,  or  in  IMC),  pilots  can  experience  SD  illusions.  Even  on  a  clear  VFR  day,  the  eyes  can 
“play  tricks”  (e.g.,  runway  and  approach  visual  illusions). 

The  inability  of  pilots  to  accurately  and  intuitively  perceive  aircraft  position  without  reliance 
upon  visual  cues  (from  flight  instruments  or  the  outside  world)  is  a  major  crux  of  the  aviation 
mishap  problem.  Maintaining  spatial  orientation  cannot  be  done  in  present-day  flight  operations 
unless  one  is  attending  to  the  appropriate  visual  cues.  Unfortunately,  many  of  an  aviator’s 
distracting  secondary  flight  tasks  are  also  of  a  visual  nature,  so  continuous  attention  to  one’s 
spatial  orientation  cannot  be  maintained  using  current  visual  displays.  Furthennore,  attention 
can  also  be  diverted  to  non-visual  sources  as  well  (Wickens,  2002),  for  example  the  distraction 
of  voice  communications  or  problem  solving.  Holmes  et  al.  (2003)  identify  “distraction/task 
saturation”  as  an  important  factor  in  their  list  of  causes  of  spatial  disorientation.  The  problem 
concerning  the  allocation  of  limited  attentional  resources  is  compounded  by  the  fact  that  atten- 
tional  resources  will  be  drawn  to  more  natural  and  salient  body  (vestibular)  cues  concerning 
orientation,  which  in  the  environment  of  flight  are  not  veridical.  In  other  words,  the  problem  of 
SD  in  flight  is  not  caused  merely  by  attentional  limitations;  rather,  the  problem  is  the  formation 
of  an  incorrect,  yet  persuasive,  subconscious  tendency  to  rely  upon  vestibular  orientation  cues. 
The  typical  SD  mishap  occurs  when  visual  attention  is  distracted  from  the  aircraft’s  orientation 
instruments  and  the  horizon  is  not  visible  or  not  being  monitored  (McGrath,  2000). 

Because  a  realistic,  achievable  approach  to  modeling  SD  requires  understanding  and  modeling 
the  vestibular  system,  the  following  section  highlights  the  human  vestibular  system  and  its 
behaviors  during  fixed-wing  aircraft  flight. 


2.4  Vestibular  System 

The  parts  of  the  vestibular  system  on  which  we  focus  are  the  semicircular  canals,  and  the  otoliths 
or  vestibular  sacs  in  the  inner  ear  (Figure  3).  The  semicircular  canals  are  filled  with  fluid  that 
indicates  bi-directional  rotation  in  the  pitch,  roll,  and  yaw  axes  via  the  cilia,  or  hair  cells,  within 
each  canal.  The  canals  are  actually  accelerometers  that  sense  changes  in  velocity.  Figure  4 
shows  the  role  of  the  semicircular  canals  in  the  three  dimensions  of  flight.  The  otoliths,  connected 
to  the  base  of  each  semicircular  canal,  respond  to  linear  accelerations  and  decelerations. 

As  rotational  motion  occurs,  the  cilia  (hair  cells)  in  the  canals  bend  in  response  to  the  relative 
motion  between  the  canal  walls,  to  which  the  cilia  are  attached,  and  the  fluid  within  the  canal.  The 
fluid’s  inertia  provides  initial  resistance  and  bends  the  cilia,  which  informs  the  brain  of  the 
acceleration  axis  and  the  magnitude  of  angular  acceleration.  After  undergoing  sustained  and 
constant  angular  motion,  however  (e.g.,  continuous  heading  change  in  a  standard  rate  turn),  the  fluid 
within  the  canals  begins  to  rotate  at  a  velocity  corresponding  to  the  body  and  head.  As  a 
consequence,  there  is  no  longer  relative  motion  between  the  cilia  and  the  fluid  within  the  canals,  the 
cilia  are  not  bent,  and  so  no  sensation  of  motion  is  sent  to  the  brain.  This  phenomenon  is  called 
“washout”  and  has  important  implications  for  spatial  disorientation.  This  washout  occurs  after 
approximately  10-15  seconds  of  sustained  rotary  acceleration  (Gillingham  &  Previc,  1993). 
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It  is  important  to  note  that  there  is  individual  variability  for  washout  timing,  as  well  as  for  the  initial 
detection  of  motion  change,  described  next.  This  variability  applies  between  people  and  within  an 
individual  under  varying  circumstances.  It  is  this  variability  that  makes  modeling  the  vestibular 
system  very  challenging,  which  complicates  the  detection  component  of  our  approach.  And,  one 
can  imagine  that  if  the  detection  component  is  difficult,  then  the  aiding  portion  is  even  more  so. 


Figure  3.  The  structure  of  the  outer,  middle  and  inner  ears  showing  the  location  of  the  semicircular  canals 

(Rosenzweig,  et  al. ,  2002,  pg.  67) 


Figure  4.  The  role 

Another  important  tenn  to  SD  research  is  Mulder’s  Law  (DeHart  &  Davis,  2002),  which  describes  a 
threshold  (called  Mulder’s  Constant)  below  which  accelerations  are  not  sensed  by  the  human 
vestibular  system.  For  an  angular  acceleration  to  be  perceived,  the  product  of  the  intensity  or 
magnitude  of  acceleration  (deg/s  )  and  time  (seconds)  of  application  must  reach  a  threshold  value. 
The  best  way  to  illustrate  the  meaning  of  Mulder’s  Law  is  with  a  few  examples,  where  Mulder’s 
Constant  is  assumed  to  be  2.5°/sec: 

1 .  If  a  person  experiences  an  acceleration  of  l°/sec“  for  1  sec,  he  or  she  will  probably  not  sense 
that  acceleration  because  the  product  (l°/sec)  does  not  exceed  Mulder’s  Constant. 

2.  If  the  same  acceleration  occurs  for  3  sec,  however,  it  will  likely  be  detected  (because  the 
product,  3°/sec,  exceeds  Mulder’s  Constant). 


of  the  semicircular  canals  in  the  perception  of  3-dimensional  motion  (US  Army,  2000). 
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3.  Even  a  large  acceleration  of  10°/sec“  will  not  be  felt,  if  its  duration  is  less  than  0.2  sec.  The 
same  acceleration  will  be  felt,  if  its  duration  is  0.25  sec  or  greater. 

It  is  important  to  note  that  not  all  humans  have  identical  thresholds  and  various  researchers  have 
detennined  slightly  different  threshold  values.  The  following  is  a  list  of  the  most  commonly 
accepted  threshold  values: 

•  Mulder  (Guedry,  1974):  2. 5 °/sec  for  all  three  axes. 

•  Stapleford  (1968):  3.2°/sec  for  roll;  2.6°/sec  forpitch;  and,  l.l°/sec  for  yaw. 

•  Oman  (2005):  1 .5°/sec  for  all  three  axes. 

While  the  semicircular  canals  sense  rotary  motion,  the  otolith  organs  sense  linear  accelerations. 
Here,  in  the  presence  of  linear  acceleration,  the  cilia  within  the  otoliths  will  bend  opposite  to  the 
direction  of  acceleration.  This  angle  of  bend,  relative  to  the  orientation  of  the  organ,  signals  the 
magnitude  of  acceleration.  The  orientation  of  the  cilia  is  affected  by  gravitational  force  as  well  as 
by  accelerations  (Howard,  1986).  Hence  a  linear  forward  acceleration  will  produce  the  same 
vestibular  sensation  as  a  tilt  backward,  producing  the  phenomenon  of  the  somatogravic  illusion 
(Gillingham  &  Previc,  1993;  Tokumaru  et  al.,  1998),  in  which  pilots  who  execute  a  rapid 
acceleration  (for  example,  on  a  missed  approach)  may  incorrectly  perceive  a  pitch-up  motion. 

As  with  the  semi-circular  canals,  the  otoliths  have  thresholds  of  detection.  In  the  Z  (vertical  axis) 
the  threshold  is  0. 1 5  m/s“;  for  both  X  (roll)  and  Y  (pitch)  axes  the  threshold  is  0.06  m/s  (Previc  & 
Ercoline,  2004,  pg.  54).  As  with  rotational  motions,  we  assume  there  is  individual  variability. 


2.5  Phase  II  Highlights 

Starting  with  a  solid  foundation  from  Phase  I  in  tenns  of  theory  and  a  working  prototype,  we 
began  Phase  II  with  a  primary  goal  of  validating  our  models,  via  experiments,  and  adjusting  the 
models  as  needed  to  incorporate  experimental  results,  as  well  as  the  new  knowledge  gained  from 
collaborators  and  from  the  literature.  Some  experiments  proceeded  as  planned;  others  experi¬ 
enced  minor  or  major  glitches.  Regardless  of  the  execution  details,  we  made  the  most  of  the 
resulting  data.  Without  doubt,  however,  the  most  valuable  Phase  II  experiences  were  when  we 
analyzed  actual  flight  data  from  military  and  civilian  accidents  where  SD  was  believed  to  be  a 
contributing  factor.  Section  3.5  describes  our  analytical  process  and  summarizes  the  results.  Of 
the  12  data  sets  analyzed,  SDAT  results  agreed  with  the  safety  investigation  findings  for  six; 
three  were  inconclusive;  two  disagreed;  and  one  data  set  was  corrupted  and  unusable.  For  the 
two  where  SDAT  results  disagreed  with  the  official  findings,  additional  knowledge  about  the 
nature  of  the  flights  would  have  suggested  that  SDAT  was  inappropriate  to  use  because  those 
flights  involved  disorientation  not  related  to  vestibular  illusions. 

We  did  not  accomplish  as  much  with  SAVVOY  as  we  had  hoped.  Due  to  our  emphasis  on  SD 
detection,  the  aiding  portion  of  our  R&D  necessarily  received  less  emphasis.  SAVVOY 
influences  SD  countermeasures  because  SOAS  favors  modalities  that  are  not  (or  less)  loaded 
than  others.  Despite  this  shortcoming,  we  conducted  two  experiments  whose  focus  was  on 
multisensory  countenneasures. 
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About  halfway  through  Phase  II,  we  realized  that  some  of  our  innovative  work  might  be 
patentable.  After  obtaining  pennission  from  our  customer  and  her  contracting  experts,  we  hired 
local  patent  attorneys.  The  result  is  our  US  patent  application  at  Appendix  D.  As  of  this  writing, 
we  have  received  no  feedback  about  our  application  from  the  Patent  and  Trademark  Office. 
When  we  do,  we  shall  keep  our  customer  apprised  of  significant  developments. 

We  designed  and  conducted  two  other  experiments  whose  primary  focus  was  on  detection.  All 
of  the  experiments  are  explained  later,  with  emphasis  on  how  we  used  the  results.  In  the  case  of 
the  detection-oriented  experiments,  results  directly  impacted  our  SDAT/SOAS  models.  For  each 
formal  experiment  we  submitted  a  journal  article.  Those  articles  (some  accepted,  some  still  in 
review)  are  in  Appendix  A. 

Furthermore,  we  presented  the  highlights  of  our  research  at  several  professional  conferences. 

The  first  was  HCI-Aero  in  Toulouse  in  June  2004;  next  was  WinterSim  in  DC  in  December 
2004;  following  was  the  AIAA-ATIO  conference  in  September  2005;  the  last  was  AsMA  in  May 
2006  in  Orlando.  We  also  exhibited  at  AsMA  in  May  2005  in  Kansas  City.  It  was  there  where 
we  met  many  of  the  AFSC  people  with  whom  we  would  later  collaborate  on  SD  mishap 
analyses.  (We  also  attended  AsMA  2004  in  Anchorage,  but  that  was  between  Phase  I  and  Phase 
II  and  so  not  under  contract  funding.)  The  final  noteworthy  publication-related  item  is  an  article 
published  in  a  Russian  aerospace  bulletin  in  the  April-May  and  June- July  2005  issues;  it  is  a 
lengthy  abstract  from  our  Phase  I  report  translated  into  Russian  by  the  editor  and  his  staff. 
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3  Accomplishments 


Re-visiting  the  dual  theme  of  detection  and  aiding,  the  following  sections  of  this  chapter  start 
with  detection,  and  then  address  aiding.  First  we  explain  our  modeling  approach  (Section  3.1), 
then  our  aiding  philosophy  (Section  3.2),  followed  by  the  experiments  we  conducted  to  support 
each  component  of  the  detection-aiding  challenge  (Section  3.3).  After  that  are  sections 
describing  our  software  (3.4)  and  our  SD  mishap  data  analyses  (3.5). 

3.1  Modeling 

Modeling  focuses  mainly  on  SD  detection,  although  some  workload  modeling  determines 
countenneasure  triggering.  The  pilot  workload  modeling  is  much  more  simplified  than 
originally  envisioned  at  the  start  of  Phase  II.  It  can  best  be  summarized  by  stating  that  SOAS 
suppresses  audio  countermeasures  when  the  pilot’s  audio  workload  is  high.  Other  modeling 
relates  to  triggering  countermeasures  based  upon  the  state  of  the  aircraft,  such  that  more  intrusive 
countermeasures  trigger  when  the  situation  is  more  dire  (e.g.,  imminent  crash).  First,  we  present 
extensive  detail  of  our  detection  modeling  in  the  following  sub-sections. 

Spatial  Disorientation  Detection 

One  of  our  goals  for  this  work  has  always  been  to  support  the  pilots  during  actual  flight.  As 
such,  the  system  for  detecting  spatial  disorientation  has  to  function  with  the  information 
available  within  the  cockpit.  Likewise,  the  models  of  spatial  disorientation  were  limited  to 
physiology  that  responded  to  aircraft  motion.  Therefore,  the  SD  detection  system  is  a 
combination  of  aircraft  flight  dynamics  and  vestibular  perception  models. 

Figure  5  shows  the  high  level  process  for  the  detection  system.  The  vestibular-based  models 
predict  the  perception  of  roll  axis  and  yaw  axis  (heading)  rotation  as  a  function  of  semi-circular 
canal  response  to  actual  aircraft  angular  accelerations.  Pitch  angle  perception,  using  an  otolith 
model,  is  based  on  the  aircraft’s  longitudinal  linear  accelerations.  The  SD  illusion  models  use  a 
combination  of  flight  dynamics  and  perceived  attitude  values  to  determine  when  a  pilot  may  be 
experiencing  a  specific  SD  illusion  (i.e.,  Leans,  Graveyard  Spiral,  Coriolis). 


Figure  5.  SD  detection  system  flow. 
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The  purpose  of  the  vestibular  model  is  to  predict  the  attitude  cues  provided  to  the  pilot  by  the 
vestibular  system  in  the  absence  of  visual  orientation  cues.  The  model  uses  basic  aircraft  flight 
dynamic  data  (available  to  accident  investigators,  as  well  as  aboard  modern  aircraft  in  real-time 
within  the  avionics  system)  and  the  associated  responses  of  the  vestibular  system.  The  angular 
rotation  model  uses  Mulder’s  Law  and  perception  threshold  values  to  detennine  which  roll  and 
yaw  angular  accelerations  are  perceived  by  the  pilot.  The  concept  of  washout  is  used  to  predict 
the  loss  of  the  sense  of  rotation.  The  pitch  perception  model  uses  linear  accelerations  to 
calculate  the  pitch  information  provided  by  the  otoliths.  Data  values  at  a  fairly  high  sampling 
rate  (4-8  Hz)  for  roll  and  pitch  angles  are  commonly  available  from  modem  aircraft.  For  the 
purpose  of  this  work,  we  use  the  term  “yaw  rotation”  to  denote  the  rotation  of  the  aircraft  as  its 
heading  changes.  For  each  time  period,  the  yaw  rate  and  acceleration  are  calculated  as  a  function 
of  the  change  in  heading. 

Initial  Rotation  Perception  -  Implementing  Mulder’s  Law 

Mulder’s  Law  (Guedry,  1974)  combines  the  magnitude  of  an  angular  acceleration  and  the  time 
of  application  to  determine  if  a  change  in  the  attitude  of  the  aircraft  is  sufficiently  high  to  be 
perceived  by  the  vestibular  system.  In  essence,  larger  accelerations  over  short  periods  of  time 
and  smaller  accelerations  over  long  periods  of  time  can  both  be  perceived.  Applying  Mulder’s 
Law  to  any  single  time  slice  of  flight  data  is  relatively  simple.  The  result  of  multiplying  the 
rotational  acceleration  by  the  time  interval  is  compared  to  the  threshold  value.  If  the  product  is 
above  Mulder’s  Constant  (where  we  use  Oman’s  (2005)  recommendation  of  1.5°/sec  for  each 
axis),  the  change  in  attitude  is  perceived  by  the  vestibular  system.  This  value-by-value 
implementation  is  referred  to  as  “instantaneous  Mulder’s.” 

Table  1  shows  yaw  data  sampled  at  approximately  1  Hz  and  the  resulting  instantaneous  Mulder’s 
calculation.  The  reader  will  notice  that  there  is  a  change  in  rotation  (a  left  turn)  starting  at 
around  33  seconds  (line  4)  and  that  the  turn  rate  stabilizes  by  about  46  seconds  (line  16).  Also 
notice  that  none  of  the  instantaneous  Mulder’s  calculations  reach  the  1.5  threshold  level.  Using 
the  instantaneous  calculation,  we  would  predict  that  a  pilot  deprived  of  visual  input  would  likely 
not  feel  the  acceleration  into  this  turn  (if  it  was  coordinated;  i.e.,  no  slip  or  skid).  The  result 
would  be  an  approximately  3.5  deg/s  discrepancy  (delta)  between  actual  and  perceived  yaw 
rotation. 

A  more  complex  implementation  is  required  to  model  Mulder’s  Law  calculations  for  durations 
larger  than  the  sampled  data  interval.  Again,  the  idea  is  that  small  accelerations  can  be  perceived 
if  they  occur  for  a  long  enough  time.  Our  implementation  involves  averaging  multiple 
acceleration  values  and  then  multiplying  by  the  time  duration.  For  example: 

Lines  4  &  5  have  accelerations  of  -1.0  and  -.85 

The  average  is  -0.93  deg/sec 

The  time  interval  of  application  is  34.6  -  32.42  =  2. 18  seconds 

The  Mulder’s  Law  calculation  is  then  0.93  *  2.18  =  2.03 
This  combined  acceleration  is  greater  than  the  1.5  threshold  so  the  model  predicts  that  the 
corresponding  change  in  velocity  over  this  time  period  would  be  perceived.  The  amount  of  that 
change  is  calculated  by  the  difference  in  velocity  across  the  associated  lines  of  data  (2.08  -  0.06 
=  2.02  deg/s). 
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Table  1.  Yaw  data  at  a\ 

pproximately  1  Hz  sam 

pling  rate. 

Line  # 

Time 

Heading 

Yaw 

Rate 

Yaw 

Acceleration 

Instantaneous 

Mulder's 

1 

30.25 

200.4 

-0.01 

0.0 

0 

2 

31.34 

200.4 

0 

0.01 

0.01 

3 

32.42 

200.33 

-0.06 

-0.06 

-0.06 

4 

33.52 

199.05 

-1.16 

-1.0 

-1.1 

5 

34.6 

196.8 

-2.08 

-0.85 

-0.92 

6 

35.66 

193.96 

-2.68 

-0.57 

-0.6 

7 

36.75 

191.22 

-2.51 

0.16 

0.17 

8 

37.85 

188.26 

-2.69 

-0.16 

-0.18 

9 

38.95 

185.52 

-2.49 

0.18 

0.2 

10 

40.03 

182.68 

-2.63 

-0.13 

-0.14 

11 

41.12 

178.72 

-3.63 

-0.92 

-1 

12 

42.2 

173.92 

-4.44 

-0.75 

-0.81 

13 

43.3 

168.88 

-4.58 

-0.13 

-0.14 

14 

44.4 

165.01 

-3.52 

0.96 

1.06 

15 

45.5 

161.83 

-2.89 

0.57 

0.63 

16 

46.59 

158.04 

-3.48 

-0.54 

-0.59 

17 

47.68 

154.27 

-3.46 

0.02 

0.02 

18 

48.78 

150.45 

-3.47 

-0.01 

-0.01 

19 

49.89 

146.61 

-3.46 

0.01 

0.01 

20 

50.97 

142.88 

-3.45 

0.01 

0.01 

21 

52.08 

139.03 

-3.47 

-0.02 

-0.02 

One  problem  with  this  method  is  to  know  when  to  stop  combining  accelerations.  According  to 
Previc  and  Ercoline  (2004)  Mulder’s  Law  holds  true  for  accelerations  less  than  10  seconds  in 
duration.  As  such,  the  model  limits  the  number  of  combined  accelerations  to  a  10-second  period. 
However,  the  model  does  not  simply  combine  each  10-second  interval  for  a  result.  For  example, 
combining  very  low  accelerations  (lines  1-3)  with  larger  ones  (lines  4-5)  could  result  in  masking 
perceived  changes.  Our  answer  is  to  check  the  combination  of  each  successive  acceleration,  in 
turn,  to  detennine  if  it  reaches  the  threshold  value.  The  logic  proceeds  as  follows: 


Given  any  data  sequence: 

■  If  Instantaneous  Mulder’s  (IM)  of  line  1  is  above  threshold,  then  apply  the 
change  to  the  perceived  attitude  value  and  go  to  next  line  of  data. 

■  If  IM1  is  below  threshold,  then  do  not  apply  any  change  to  the  perceived 
attitude  value  and  go  to  next  line  of  data. 

■  If  IM2  is  above  threshold,  then  apply  the  change  from  2;  ignore  1  and 
increment. 

■  If  IM2  is  below,  then  check  the  combination  of  1&2. 

■  If  combined  1&2  is  above,  then  apply  the  change  and  increment. 

■  If  combined  1&2  is  below,  then  do  not  apply  a  change;  check  the  next  line. 


Extending  this  sequence  to  a  long  series  of  sub-threshold  individual  and  combined 
accelerations,  the  calculation  sequence  is: 
check  line  1 
check  line  2 
check  2  &  1 
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check  3 
check  3  &  2 
check  3  &  2  &  1 
check  4 
4,3 
4,3,2 
4,3,2,  1 
etc. 

This  process  allows  the  model  to  check  each  subsequent  acceleration  without  unnecessarily 
combining  it  with  very  low  accelerations  that  might  mask  a  perceived  acceleration  while  still 
checking  long  sequences  of  low  accelerations.  If  combinations  of  up  to  10  seconds  are  checked 
without  exceeding  the  detection  threshold,  the  algorithm  drops  the  first  value  in  the  sequence  and 
includes  the  next  one.  This  way  it  acts  as  a  10-second  sliding  window  of  perception. 

However,  using  this  process  could  yield  computations  that  assess  too  many  combinations,  thus 
jeopardizing  real-time  perfonnance  (which  is  a  huge  concern  for  SOAS,  less  so  for  SDAT).  If 
the  sampling  rate  is  1  Hz,  then  VAC  could  check  the  sequence  for  ten  lines  of  data.  The  number 
of  calculations  is  [(n  *  (n-l))/2],  so  evaluating  10  lines  of  data  requires  45  calculations.  If  the 
sampling  rate  is  2  Hz  then  190  calculations  are  needed.  Since  aircraft  data  are  sometimes 
sampled  at  frequencies  of  8  Hz  or  more,  processing  time  for  a  cockpit  system  becomes  an  issue. 
To  shorten  the  sequence,  the  algorithm  uses  the  assumption  that  very  low  accelerations  will 
never  be  perceived  even  when  combined  for  10  seconds.  For  example,  accelerations  below  0. 1 
are  not  combined  and  the  calculation  increments  to  the  next  line  of  data. 

Rotation  Perception  Decay  Model 

The  next  step  in  the  rotation  perception  model  is  the  representation  of  washout.  During  constant 
rotation,  such  as  a  turn,  the  rotation  perception  information  from  the  semi-circular  canals 
progressively  diminishes.  The  characteristic  of  this  decay  can  be  model  using  an  exponential 
decay  function.  Our  model  uses  the  formula: 

N(t)  =  Noe4,1,  where: 

N0  is  the  starting  rotation  value; 
t  is  the  time  since  the  start  of  the  decay; 

X  is  the  decay  constant;  and,  therefore, 

N(t)  is  the  perceived  rotation  at  time  t. 


In  the  example  in  Figure  6,  the  perceived  rotation  value  has  reached  5  deg/sec.  After 
approximately  2.5  seconds  the  decay  is  approximately  3  deg/sec,  so  the  perceived  rotation  rate  is 
only  2  deg/sec.  By  about  8  seconds  the  sense  of  rotation  has  decayed  completely. 
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Figure  6.  Exponential  rotation  perception  decay  example. 

The  decay  constant  in  the  formula  is  used  to  change  the  rate  of  decay  (shape  of  the  curve). 
Vestibular  system  research  provides  a  range  of  time  over  which  complete  washout  seems  to 
occur.  For  example,  Previc  and  Ercoline  (2004)  suggest  about  10  sec,  whereas  Guedry  (1974) 
suggests  that  washout  may  take  considerably  longer:  20-30  sec.  However,  these  values  do  not 
support  the  development  of  a  decay  rate  across  different  levels  of  initial  rotation  perception.  The 
results  of  one  of  the  experiments  performed  during  this  project  (Section  3.2)  were  used  to 
calibrate  the  decay  model.  A  decay  constant  value  of .  12  in  the  decay  formula  best  represents 
the  results  of  the  decay  duration  from  this  experiment. 

The  determination  of  when  to  start  and  stop  any  particular  rotation  decay  sequence  (or  what 
constitutes  constant  rotation)  uses  the  Mulder’ s-based  acceleration  perception  model.  For  the 
purpose  of  the  model,  constant  rotation  is  defined  as  any  sequence  of  accelerations  that  are  below 
the  vestibular  detection  threshold.  If  the  Mulder’s  calculation  determines  that  an  acceleration  is 
below  threshold,  then  the  starting  value  of  rotation  for  the  decay  calculation  is  the  current 
perceived  value.  Decay  continues  until  an  above  threshold  acceleration  occurs. 

Combining  Perception  and  Decay 

Much  of  the  research  of  angular  acceleration  thresholds  seems  to  have  been  applied  from  the 
point  of  zero  rotation.  A  stationary  subject  is  accelerated  in  a  specific  angular  plane  at  different 
rates  and  the  threshold  at  which  they  sense  the  rotation  is  recorded.  We  know  of  no  study  that 
tried  to  determine  detection  thresholds  for  participants  who  start  from  a  non-zero  angular 
velocity.  It  is  possible  that  threshold  perception  values  could  be  different  when  already  rotating 
as  apposed  to  starting  from  zero  rotation  given  the  non-linearities  of  cupolary  response.  Our 
assumption  is  that  we  can  use  the  same  threshold  values  to  apply  velocity  changes  to  perceived 
values  whether  or  not  the  pilot  is  already  sensing  rotation.  The  extension  to  that  assumption  is 
that  the  velocity  change  associated  with  a  perceived  acceleration  is  additive  to  the  current 
perceived  value. 

The  vestibular  perception  model  combines  the  implementation  of  Mulder’s  Law  with  the  decay 
sequence  to  calculate  the  different  between  the  actual  aircraft  angular  rotation  and  the  angular 
rotation  information  provided  to  the  pilot  through  the  vestibular  system.  Figure  7  shows  the 
analysis  sequence  for  the  semi-circular  canal  model  for  the  perception  of  rotation  in  yaw  and  roll. 
If  a  specific  acceleration  is  above  threshold  then  the  same  velocity  change  applied  to  the  aircraft 
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is  applied  to  the  perceived  rotation  value  (i.e.,  it  is  noticed).  If  a  specific  acceleration  is  below 
threshold  then  the  associated  small  change  to  the  aircraft  rotational  velocity  is  not  applied  to  the 
perceived  velocity  value  resulting  in  a  difference  between  the  actual  aircraft  (AC)  yaw  or  roll 
rotation  rates  and  the  perceived  value  of  those  rotations.  For  any  sub-threshold  change,  the 
washout  calculation  is  then  applied  to  predict  how  much  of  the  sense  of  rotation  is  lost  for  the 
given  time  period.  For  each  time  period,  the  model  then  calculates  the  delta  (difference)  between 
the  actual  and  perceived  attitudes. 


Figure  7.  Perceived  attitude  algorithm  sequence. 


Actual  versus  Perceived  Rotation  Delta  Examples 

The  following  two  examples  show  rotation  perception  deltas  predicted  by  the  vestibular  model 
when  applied  to  real  flight  data.  The  reader  should  recall  that  the  perceived  rotation  values  are 
based  on  the  information  provided  by  the  vestibular  system  without  any  visual  information. 

The  graph  in  Figure  8  shows  an  example  of  actual  versus  perceived  rotation  as  calculated  by  the 
vestibular  model.  The  data  set  is  approximately  three  minutes  long  and  shows  a  long  left  turn. 
The  blue  line  in  the  figure  show  that  the  yaw  rotation  (heading  change)  reaches  a  maximum  rate 
of  about  -5  deg/sec  (negative  values  are  left  turns).  Partway  through  the  turn,  the  rotation  drops 
to  zero  before  increasing  again.  The  rate  of  change  of  the  rotation  is  slow  enough  to  be  below 
the  perception  threshold  value  for  nearly  the  whole  data  set.  The  red  line  shows  the  rotation  rate 
as  calculated  by  our  vestibular  model.  Since  the  accelerations  are  below  threshold,  there  is 
almost  no  perception  of  the  turn  as  it  occurs.  The  only  supra-threshold  movement  is  at  about 
time  125  when  the  rotation  rate  is  quickly  reduced  by  about  2  deg/sec.  The  movement  is  above 
the  vestibular  perception  threshold  and  the  pilot  would  feel  approximately  2  deg/sec  of  rotation 
to  the  right  (illusory  rotation).  Then,  the  movement  back  to  -5  deg/sec  rotation  is  sub-threshold, 
so  it  is  probably  not  perceived  by  the  pilot.  As  such,  it  is  the  decay  (washout)  of  the  sense  of 
illusory  rotation  that  brings  the  perception  of  rotation  back  to  zero  a  few  seconds  later.  The 
magnitude  of  perceived  illusory  rotation  is  determined  by  the  magnitude  and  duration  of  the 
original  actual  rotation,  as  well  as  the  rate  of  the  deceleration  from  that  rotation,  an  issue 
examined  in  an  experiment  described  in  Section  3.3. 
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Figure  8.  Actual  vs.  perceived  yaw  rate  example  1. 


Figure  9  shows  another  example  of  actual  versus  perceived  yaw  rotation.  Over  about  2  minutes, 
the  blue  line  shows  a  series  of  three  right  hand  turns.  The  rotation  rate  of  the  first  turn  is 
maintained  at  about  2  deg/sec.  The  red  line,  representing  the  perceived  rotation  rate,  shows  that 
the  initial  acceleration  at  about  time  23  is  perceived  but  then  washes  out  to  zero  perceived 
rotation  by  about  time  40.  The  second  turn  is  at  a  maximum  of  about  8  deg/sec  rotation  and 
again  the  perception  of  rotation  is  gradually  lost  during  the  turn.  When  the  actual  rotation 
returns  to  zero  (after  rolling  out  of  the  turn),  the  rotational  deceleration  is  above  threshold 
resulting  in  over  6  deg/sec  of  illusory  rotation  in  the  opposite  direction.  The  final  -4  deg/sec  of 
perceived  illusory  rotation  washes  out  starting  from  about  time  100  and  ending  at  about  time  116 
while  the  actual  rotation  value  is  about  zero  for  the  same  time  period. 


Figure  9.  Actual  vs.  perceived  yaw  rate  example  2. 
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Pitch  Perception  Model 

The  perception  of  pitch  angle  is  dominated  by  another  organ  of  the  vestibular  system  call  the 
otoliths.  Unlike  the  semi-circular  canals,  otoliths  respond  to  linear  accelerations.  We  are 
particularly  interested  in  the  Gx  (longitudinal)  and  Gz  (vertical)  sensing  otoliths,  since  those  are 
the  accelerations  most  commonly  experienced  by  fixed-wing  aircraft  pilots.  In  nonnal  ground 
activity,  this  sense  is  dominated  by  the  acceleration  due  to  gravity,  since  most  self-generated 
human  motion  is  significantly  less  than  1  G.  Aircraft,  however,  can  create  both  linear  and 
rotational  accelerations  that  are  much  higher.  The  otoliths  essentially  respond  the  acceleration 
vectors.  The  combination  of  aircraft  acceleration  and  gravitational  vectors  can  cause  the  otoliths 
to  create  an  incorrect  perception  of  which  way  is  down.  This  is  most  apparent  in  the  sensation  of 
pitch.  A  combination  of  1  G  gravitational  vector  (Gz)  and  1  G  linear  acceleration  vector  (Gx), 
results  in  the  perception  of  a  45  degree  pitch  up  angle  (Figure  10). 


Figure  10.  Acceleration  vectors  and  pitch  perception  illusion 
(adapted  from  DeHart  and  Davis,  2002). 

Vector  analysis  is  used  to  combine  the  linear  acceleration  with  the  acceleration  due  to  gravity  to 
arrive  at  the  perceived  pitch  angle  using  the  formula:  Perceived  Pitch  =  Actual  Pitch  + 
ArcTangent(Gx/Gz)  (DeFIart  &  Davis,  2002). 

For  our  calculations,  Gx  is  linear  acceleration  calculated  from  the  change  in  airspeed  and 
converted  to  G.  Gz  is  taken  from  the  Normal  Load  Factor  (or  similar)  value  in  the  data  set.  By 
definition,  Gx  and  Gz  are  orthogonal.  Figure  1 1  shows  an  example  of  actual  versus  perceived 
pitch  based  on  the  model  calculation.  This  data  set  starts  with  a  takeoff  so  the  actual  pitch  is  0 
until  above  time  120,  then  changes  quickly  to  about  12  degrees.  The  high  linear  acceleration 
associated  with  takeoff  causes  a  feeling  of  even  greater  pitch-up  during  that  time  period.  While 
the  rest  of  the  data  set  shows  a  few  smaller  sustained  differences  and  the  occasional  high  spike, 
none  of  the  differences  are  as  great  as  during  the  takeoff  -  an  expected  result. 
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This  calculation  may  only  accurate  when  the  aircraft  is  banked  at  30  deg  or  less.  More  than  that 
and  the  perception  of  ‘down’  is  combined  with  the  bank  to  the  side  and  the  pure  pitch  calculation 
may  no  longer  be  valid.  Although  VAC  still  computes  perceived  pitch  at  larger  bank  angles  -  at 
all  bank  angles,  in  fact  -  care  must  be  taken  in  interpreting  the  results  under  these  conditions. 


Spatial  Disorientation  Illusion  Sequence  Models 

We  know  of  no  way  to  determine  the  actual  intentions  or  perceptions  of  the  pilot.  As  such,  it  is 
difficult  to  detennine  if  a  specific  SD  illusion  was  experienced  based  only  on  flight  dynamics. 
However,  looking  for  specific  sequences  within  a  data  set  and  combining  those  with  the 
perception  predictions  from  the  vestibular  model,  can  give  us  a  general  indication  that  an  SD 
problem  might  be  occurring.  Three  illusion  models  (Leans,  Graveyard  Spiral,  and  Coriolis)  are 
currently  used  to  look  for  such  sequences.  They  assume  that  there  is  no  visual  information 
available  to  the  pilot,  or  that  the  pilot  is  not  attending  to  such  information.  The  models  are  a 
sequence  of  events  that  must  occur  in  a  specific  order  with  specific  values  and  time  periods.  As 
each  event  occurs  the  certainty  of  SD  increases. 

The  reasons  for  selecting  these  three  illusions  are  the  same  as  in  Phase  I.  Briefly,  Leans  is  the 
most  common  illusion  experienced  by  pilots,  is  insidious,  and  is  often  a  precursor  to  more  severe 
illusions  such  as  Graveyard  Spiral  or  Spin.  Graveyard  Spiral  is  another  common  SD  illusion  and 
lends  itself  well  to  modeling  and  experimental  validation  (Section  3.3).  Coriolis  is  a  sudden 
incapacitating  illusion  that  seemed  worthwhile  to  address,  except  for  the  fact  that  very  few  flight 
data  sets  contain  the  pilot’s  head  position  data. 

The  Leans  Illusion  Model 

The  most  common  SD  illusion  in  flight  is  the  Leans  (Holmes  et  ah,  2003;  Benson,  1988),  which 
entails  an  erroneous  feeling  of  roll.  A  typical  case  occurs  in  the  following  scenario  (Gillingham 
&  Previc,  1993):  In  instrument  flight  (horizon  not  visible),  the  pilot  has  very  slowly  entered  a 
turn,  perhaps  unknowingly,  at  a  sub-threshold  rate  so  that  the  semicircular  canals  provide  no 
sense  of  rotation  and  the  gravito-inertial  force  is  directly  down  “through  the  seat  pan,” 
perpendicular  to  the  wings  (so  there  is  no  sense  of  being  tilted).  If  the  pilot  then  becomes 
consciously  aware  of  the  aircraft’s  bank  angle  (e.g.,  by  looking  at  the  instruments),  and 
intentionally  returns  the  bank  to  the  true  level  attitude,  he  or  she  will  now  receive  a  vestibular 
sensation  of  an  opposite  bank.  If  the  pilot  continues  to  rely  upon  flight  instruments  to  maintain  a 
level  attitude,  he  or  she  may  also  lean  in  the  orientation  of  the  incorrectly  perceived  upright 
(hence  the  illusion’s  name).  If,  instead,  the  pilot  does  not  rely  upon  the  instruments  at  this  point, 
but  rather  the  intuitive  (vestibular)  signal  of  upright,  he  or  she  will  return  the  aircraft  to  its 
original  misperceived  bank  angle.  Without  awareness  and  conscious  correction,  the  bank  can 
lead  to  a  gradual  pitch  down  attitude,  a  loss  of  altitude,  and  an  increase  in  airspeed.  A  side-to- 
side  seesawing  process  of  correction  and  re-entry  may  also  ensue,  along  with  the  loss  of  altitude, 
until  the  pilot  is  so  disoriented  that  recovery  to  straight-and-level  flight  is  difficult,  if  not 
impossible.  Such  a  scenario,  in  its  worst  form,  ends  with  a  fatal  crash. 

The  model  of  the  Leans  is  expressed  as  a  timed  sequence  of  events  with  the  certainty  of  the 
assessment  of  the  disorientation  increasing  with  each  successive  event  as  shown  in  Figure  12. 
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The  first  event  is  the  initiation  of  a  roll  at  a  rate  below  the  vestibular  threshold  starting  from  a 
specified  roll  angle  and  rate.  The  second  event  is  a  roll  angle  change  of  at  least  5  degrees  that 
lasts  at  least  5  seconds  that  is  reached  through  continuous  sub-threshold  movements.  If  these 
two  events  occur  in  sequence,  it  is  possible  that  the  pilot  has  not  noticed  the  ensuing  roll  angle 
and  that  there  is  a  difference  between  the  pilot’s  perceived  attitude  and  the  true  attitude  of  the 
aircraft.  As  such,  the  model  indicates  a  possibility  of  SD  but  only  at  a  very  low  confidence  level. 
The  third  event  is  the  loss  of  altitude  as  measured  by  negative  vertical  velocity.  If  this  event 
follows  the  first  two,  it  is  possible  that  the  pilot  has  also  not  intended  or  noticed  the  loss  of 
altitude  and  the  model  represents  an  increased  confidence  in  its  assessment  of  the  Leans.  The 
fourth  event  is  a  roll  well  above  the  vestibular  threshold  (e.g.,  greater  than  5  deg/sec).  If  this 
occurs  following  the  first  three  events,  it  is  possible  that  the  pilot  has  now  noticed  the  roll  angle 
and  has  quickly  corrected  back  toward  level.  When  this  occurs,  the  pilot’s  vestibular  system  will 
register  a  roll  to  an  opposite  bank  angle,  again  resulting  in  a  difference  between  the  perceived 
attitude  and  the  actual  attitude  of  the  aircraft.  At  this  point  the  model  represents  a  high  level  of 
SD  certainty.  The  final  event  in  the  model  is  the  tilt  of  the  pilot’s  head  opposite  the  perceived 
roll  angle.  If  this  occurs  following  the  other  four  events,  it  is  likely  that  the  pilot  is  experiencing 
the  Leans  and  the  model  represents  an  even  higher  level  of  SD  certainty.  Even  without  pilot 
head  position  data  for  the  fifth  event  in  the  sequence,  this  model  of  the  Leans  is  very  good  at 
detecting  likely  Leans  illusions  in  actual  flight  data. 


(1)  Sub-threshold  roll 


r  =  roll  angle 
r'  =  roll  rate 
w  =  vertical  velocity 
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Figure  12.  Leans  illusion  model  sequence. 


Graveyard  Spiral  Illusion  Model 

A  somatogyral  illusion  is  “a  false  sense  of  rotation  (or  absence  of  rotation)  that  results  from 
misperceiving  the  magnitude  or  direction  of  an  actual  rotation”  (Previc  &  Gillingham,  1993,  p. 
59).  One  such  illusion  is  the  Graveyard  Spiral  in  which  a  misperception  of  a  turn  results  in  a 
descending  and  often  tightening  spiral.  The  common  sequence  involves  a  turn  that  is  held  long 
enough  to  lose  the  sense  of  rotation  (i.e.,  washout).  The  return  to  zero  bank  and  therefore 
straight  flight,  then  feels  like  a  turn  in  the  opposite  direction.  The  pilot  then  erroneously  attempts 
to  reduce  the  sense  of  illusory  rotation  by  rolling  back  into  the  original  turn,  typically  without 
increasing  pitch  to  maintain  altitude.  If  the  pilot  then  attempts  to  arrest  the  loss  of  altitude  by 
pulling  back  on  the  stick  without  decreasing  the  bank  angle,  the  turn  tightens  and  results  in  a 
descending  spiral. 
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The  Graveyard  Spiral  illusion  model  focuses  on  the  difference  (we  use  the  term  “delta”)  between 
perceived  yaw  rotation  as  calculated  by  the  vestibular  model  and  actual  yaw  rotation.  Figure  13 
shows  the  model  event  sequence.  Event  1  looks  for  a  perceived  yaw  rotation  that  is  less  than  the 
actual  yaw  rotation  by  a  specified  delta  (5  deg/sec).  When  this  situation  exists,  it  indicates  either 
washout  or  sub-threshold  yaw  rotation.  For  the  next  two  events,  any  reduction  in  the  rotational 
delta  interrupts  the  sequence  and  indicates  more  accurate  rotational  perception  on  the  part  of  the 
pilot.  Event  2  looks  for  a  perceived  yaw  rotation  that  is  greater  than  the  actual  provided  the 
rotation  delta  is  maintained.  This  indicates  that  the  pilot  has  reduced  the  actual  rotation  and  is 
experiencing  illusory  rotation.  Event  3  looks  for  a  perceived  yaw  rotation  that  is  less  than  the 
actual  rotation,  provided  the  rotation  delta  was  maintained.  This  indicates  that  the  pilot  increased 
actual  yaw  rotation  but  does  not  perceive  much  or  any  of  it.  This  is  indicative  of  an  erroneous 
roll  back  to  the  original  rotation  in  order  to  reduce  the  sense  of  illusory  rotation.  Event  4  is 
designed  to  indicate  a  loss  of  control  following  the  other  events  by  a  combination  of  high  bank 
angle  and  negative  vertical  velocity.  Event  5  looks  for  the  pitch  up  command  that  could  tighten 
the  spiral. 


Figure  13.  Graveyard  Spiral  illusion  model  sequence. 


Coriolis  Illusion  Model 

The  FAA’s  Airman’s  Information  Manual  (ASA,  1995)  describes  the  Coriolis  illusion  as  the 
“most  overwhelming  of  all  illusions  in  flight”  (page  645).  It  occurs  during  “a  prolonged 
constant-rate  turn  that  has  ceased  stimulating  the. . .  [vestibular]  system,”  (i.e.,  washout)  when 
there  is  an  abrupt  head  movement  that  creates  “the  illusion  of  rotation  or  movement  in  an  entirely 
different  axis.”  The  resulting  tumbling  sensation  is  nearly  incapacitating.  The  Coriolis  illusion 
often  results  in  the  pilot’s  loss  of  aircraft  control,  which  can  have  catastrophic  results  if  there  is 
no  other  pilot  to  assume  control.  The  power  of  the  Coriolis  illusion  to  deceive  is  further 
supported  by  the  finding  in  a  recent  survey  that  it  was  the  most  prevalent  of  those  found  in  141 
pilots  attending  a  course  at  Randolph  AFB  in  1997-98  (Sipes  &  Lessard,  1999). 

This  description  of  Coriolis  focuses  on  rotational  washout  in  the  yaw  (heading)  axis  during  a 
sustained  turn.  The  direction  of  the  head  motion  that  then  causes  the  tumbling  sensation  would 
be  primarily  in  either  the  pitch  or  roll  axis.  While  this  is  the  most  likely  Coriolis  sequence,  it  is 
certainly  possible  for  the  initiating  event  of  rotational  washout  to  occur  in  either  the  pitch  or  roll 
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axis.  The  model  of  the  Coriolis  illusion  developed  in  this  effort  looks  for  an  event  sequence 
starting  from  rotational  washout  in  any  of  the  three  axes.  However,  for  the  sake  of  simplicity, 
this  description  of  the  model  focuses  on  the  most  common  version. 

Figure  14  shows  the  model  event  sequence.  Event  1  looks  for  a  perceived  yaw  rotation  that  is 
less  than  the  actual  yaw  rotation  by  a  specified  delta  (5  deg/sec).  This  is  the  same  initiating 
event  as  that  of  the  Graveyard  Spiral  illusion  and  indicates  either  washout  or  sub-threshold  yaw 
rotation.  While  this  delta  is  maintained,  Event  2  looks  for  a  head  position  angle  of  at  least  30 
degrees  in  either  the  pitch  or  roll  axis.  The  combination  of  these  two  events  could  initiate  the 
tumbling  sensation.  Event  3  looks  for  a  loss  in  altitude  that  might  indicate  that  the  pilot  is  having 
difficulty  controlling  the  aircraft  as  a  result  of  the  first  two  events.  While  the  occurrence  of 
Event  1  may  be  quite  common  in  flight,  the  subsequence  occurrence  of  Events  2  and  3  increases 
our  certainty  that  the  pilot  may  be  experiencing  spatial  disorientation. 


(1 )  y’p  <  y’a  &  y’d  >=  5°/sec 

(2)  aha  >=  30° 
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Figure  14.  Coriolis  illusion  model  sequence. 


The  reader  should  note  in  the  previous  illusion  model  descriptions  that  increasing  certainty  level 
is  important  to  our  assessment  of  the  likelihood  of  SD.  The  main  reason  for  assessing  certainty 
is  to  trigger  the  appropriate  SD  countermeasures  -  the  aiding  portion  of  this  R&D. 


3.2  Pilot  Aiding 

The  goal  of  intelligent  multisensory  aiding  (where  SOAS  is  one  example)  is  to  provide 
countenneasures  when  pilots  need  them  most,  and,  perhaps  more  importantly,  to  avoid  using 
them  when  they  might  be  considered  annoying  or  distracting,  and  unnecessary  (if  based  upon  a 
false  inference)  rather  than  helpful.  As  there  are  many  “bells  and  whistles”  within  modem 
cockpits,  it  does  more  harm  than  good  to  provide  even  more  cues  and  alerts  to  pilots  without  a 
high  level  of  certainty  that  they  are,  indeed,  needed.  Even  though  SOAS  is  designed  to  save 
lives,  there  is  some  risk  that  it  may  put  lives  at  risk,  if  it  mis-assesses  a  situation.  Consequently, 
an  over-riding  dictum  within  our  design  is  to  do  no  hann.  This  is  especially  important  during 
early  research,  design  and  development  when  there  may  be  many  latent  bugs  within  the  software. 

There  are  three  sources  of  inputs  into  SOAS’s  decisions  to  aid  the  pilot:  SD  certainty  level, 
aircraft  state  (e.g.,  imminent  crash),  and  pilot  SAVVOY  workload  levels.  The  following  figures 
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depict  these  decision  factors  and  are  unchanged  from  Phase  I;  they  also  reflect  the  philosophy 
evolved  during  the  Pilot’s  Associate  program  (Hammer  &  Small,  1995)  and  related  Hazard 
Monitor  R&D  (Bass  et  ah,  2004)  where  the  concepts  of  remediation  (i.e.,  applying  actions  to 
remedy  a  situation)  and  avoiding  the  adverse  consequences  of  a  situation  were  paramount. 

Figure  15  is  an  overview  of  an  SD  event.  The  boxes  at  the  left  and  right  signify  assessments  of 
the  start  and  finish  of  an  in-flight  SD  event.  The  following  three  figures  amplify  the  situation 
nodes  between  the  starting  (detection)  and  the  finish  (tennination)  boxes. 


- ► 

Increasing  severity  of  SD 


Figure  15.  SD  remediation  network  -  overview. 


There  are  three  categories  of  situations  of  concern,  each  represented  with  its  own  figure 
(interested  readers  should  consult  the  Phase  I  final  report  and/or  the  listed  citations  for  further 
information): 

•  SD  Onset  situations:  Emphasis  on  prevention  (Figure  16) 

•  SD  Evolving  situations:  Emphasis  on  helping  pilot  recognize  and  recover  (Figure  17) 

•  SD  Critical  situations:  Emphasis  on  recovery  (Figure  18) 


Types  of  Aiding: 

-  Normal  displays  (HDD, 

HUD,  HMD,  low  tactile) 

-  No  audio,  olfactory  displays 


Good  Horizon 


Defining  States: 

-  Aircraft:  controlled  flight 

-  External:  good  visibility 

-  Pilot:  no  problems 


Types  of  Aiding: 

-  Enhanced  displays  (HDD, 

HUD,  HMD,  low  tactile) 

-  Low  audio 

-  No  olfactory  display 


Obscured  Horizon 


Defining  States: 

-  Aircraft:  controlled  flight 

-  External:  fair  visibility 

-  Pilot:  no  problems 


Types  of  Aiding: 

-  Enhanced  displays  (HDD, 

HUD,  HMD) 

-  Medium  tactile 

-  Low  audio 

-  No  olfactory  display 


Horizon  Gone 


Defining  States: 

-  Aircraft:  controlled  flight 

-  External:  poor  visibility 

~  Pilot:  possibly  distracted 


Figure  16.  SD  Onset  remediation  network. 
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Types  of  Aiding: 

-  Enhanced  displays  (HDD, 

HUD,  HMD) 

-  Medium  tactile 

-  Medium  audio 

-  No  olfactory  display 


SD  Trigger  Event 


Defining  States: 

-  Aircraft:  SD-inducing  maneuver 
(slow  roll,  rapid  pitch,  etc.) 

-  External:  no  combat  reason 

-  Pilot:  SD-inducing  head  position 


Types  of  Aiding: 

-  SD  icons  on  displays 

-  High  tactile 

-  Medium  audio 

-  No  olfactory  display 


Erratic  Control 


Defining  States: 

-  Aircraft:  erratic  attitude 

-  External:  no  combat  reason 

-  Pilot:  erratic  inputs 


Types  of  Aiding: 

-  SD  icons  on  displays 

-  High  tactile 

-  High  audio 

-  No  olfactory  display 


► 


Unusual  Attitude 


► 


Defining  States: 

-  Aircraft:  initial  unusual  attitude 

-  External:  no  combat  reason 

-  Pilot:  stressed 


Figure  17.  SD  Evolving  remediation  network. 


Types  of  Aiding: 

-  SD  icons  on  displays 

-  High  tactile 

-  High  audio 

-  Olfactory  display 


Types  of  Aiding: 

-  SD  icons  on  displays 

-  High  tactile 

-  Olfactory  display 

-  Audio  recovery  commands 

-  Auto-recovery,  if  commanded 


Types  of  Aiding: 

-  SD  icons  on  displays 

-  High  tactile 

-  Olfactory  display 

-  Audio  recovery  commands 

-  Auto-recovery 

-  Auto-ejection,  if  authorized 


Unusual  Attitude 


Defining  States: 

-  Aircraft:  unusual  attitude  with 
significant  airspeed  loss  or  gain 

-  External:  no  combat  reason 

-  Pilot:  significant  stress 


Impending  Crash 


Defining  States: 

-  Aircraft:  at  flight  envelope  limit 

-  External:  no  combat  reason 

-  Pilot:  "Help!"  "Recover!" 


Imminent  Crash 


Defining  States: 

-  Aircraft:  imminent  ground 
impact  or  departure  from 
controlled  flight 

-  External:  no  combat  reason 

-  Pilot:  incapacitated 


Figure  18.  SD  Critical  remediation  network. 


As  mentioned  previously,  we  were  unable  to  thoroughly  test  SOAS’s  countermeasure  logic, 
since  the  emphasis  in  Phase  II  was  necessarily  on  detection.  Modeling  pilot  multisensory 
workload  is  non-trivial,  as  is  assessing  the  true  aircraft  state  and  severity  of  the  situation.  The 
logic  behind  increasingly  intrusive  cues  and  countermeasures  is  compelling,  but  needs  testing 
with  naive  pilots. 


The  planned  progression  is  to  begin  with  a  unique  visual  cue  and  aid,  the  SD  Icon.  As  an  SD 
situation  worsens,  SOAS  would  trigger  an  audio  cue  -  a  unique  tone  to  signify  “SD  suspected.” 
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If  the  situation  continues  to  deteriorate,  a  tactile  cue  is  next.  For  example,  vibrating  all  of  a 
tactile  vest’s  tactors  would  presumably  help  get  the  pilots  attention,  with  the  implicit  message 
again  being  “SD  suspected;  check  attitude.”  A  unique  harmless  olfactory  cue  (an  aroma)  was 
postulated  as  the  next  most  intrusive.  If  cuing  yields  no  improvement  to  the  situation  and  the 
pilot  has  entered  an  unusual  attitude  that  would  result  in  a  crash  if  not  corrected,  then  active 
countermeasures  begin.  All  the  time,  the  unique  SD  Icon  is  displayed.  Added  to  it  are  voice 
commands  to  return  to  straight  and  level  flight.  When  auditory  workload  is  high,  tactile  vest 
commands  trigger.  If  the  situation  continues  to  worsen,  a  hypothetical  auto-recovery  system 
triggers  (auto-GCAS  is  one  such  system,  currently  in  use  on  the  Swedish  Grippen  C).  Asa  last 
resort,  if  the  airplane  does  not  recover  (perhaps  due  to  damage  or  a  stall)  a  hypothetical  auto¬ 
ejection  could  save  the  pilot’s  life.  For  civilian  aircraft,  there  exists  a  ballistic  recovery  system  (a 
parachute  for  the  whole  airplane)  that  would  substitute  for  the  above  auto-recovery  and  auto¬ 
ejection  countermeasures.  The  only  other  compensatory  action  triggered  by  SO  AS  (hypotheti¬ 
cally)  is  another  olfactory  one  -  a  smelling  salts  odor  for  use  during  Coriolis  illusions  to  help  the 
pilot  “snap  out  of’  any  resulting  incapacitation. 

Developing  and  testing  a  SOAS  that  employed  all  the  above  cues  and  countermeasures  are 
important  next  steps  (Small  &  Bass,  2000),  but  exceeded  the  scope  of  Phase  II  as  we  emphasized 
getting  the  detection  portion  of  SDAT/SOAS  in  satisfactory  condition.  As  the  reader  can  see 
from  the  above  explanations,  triggering  the  wrong  cues  or  countenneasures  or  triggering  them  at 
the  wrong  time  is  worse  than  doing  nothing  in  our  view.  That  is  why  detection  had  to  take 
priority. 


3.3  Experimentation 

We  designed  and  conducted  four  experiments  in  Phase  II  (Figure  19);  three  were  fonnal  IRB- 
approved  experiments;  one  was  an  informal  flight  test  to  generate  data  for  the  models,  rather  than 
to  understand  human  performance  per  se.  The  flight  test  and  a  somatogyral  experiment  in  the 
Air  Force  Academy’s  General  Aviation  Training  (GAT)  simulator  supported  the  detection  part  of 
our  work.  An  SD  Icon  experiment  and  a  SOAS  test  (in  the  Academy’s  stationary  F-16 
simulator)  supported  the  aiding  portion  of  our  research. 


Figure  19.  Rationale  for  Phase  II  experiments. 
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Detection  Focused  Experiments 

The  GAT  motion-based  simulator  helped  explore  the  somatogyral  illusion  in  a  controlled 
environment.  Each  trial  spun  volunteer  cadets  at  one  of  three  different  speeds  for  one  of  three 
different  durations  (Figure  20).  One  goal  was  to  fill  some  knowledge  base  gaps  by  obtaining 
quantified  subjective  perceptions  of  washout  and  illusory  rotation  for  spins  in  a  range  that 
approximated  fighter  aircraft  yaw  axis  (heading)  rotation  rates  for  normal  turns,  as  well  as 
accelerate  Graveyard  Spiral  or  Spin  conditions. 

The  results  helped  us  fine-tune  our  washout  calculation  (i.e.,  exponential  decay  time  constant) 
and  the  perception  of  the  magnitude  of  illusory  rotations  upon  stopping  the  original  actual 
rotation.  For  details,  reader  should  see  the  ASEM  manuscript  in  Appendix  A. 


Experimental  Design 


Time  (seconds) 


Figure  20.  Spin  plots  for  profiles  at  20,  70,  and  120  degrees/second  for  10,  20,  and  40  seconds  with  10 
degrees/second2  ramp-up  and  ramp-down  accelerations. 


The  goal  of  the  flight  test  was  to  subject  two  volunteers  (one  female,  one  male)  to  the  Feans  and 
other  SD-inducing  maneuvers  while  recording  their  quantitative  estimate  of  their  perceptions  of 
roll  and  pitch.  The  subjects  were  blind- folded  while  a  safety  pilot  flew  10  different  maneuvers. 
The  subjective  perception  recording  device  operated  flawlessly;  unfortunately,  the  GPS-based 
flight  data  recorder  did  not.  Therefore,  it  was  difficult  to  perform  a  quantitative  comparison  and 
further  validate  or  calibrate  our  SD  models.  The  results  of  the  informal  flight  test  are  in  Table  2. 
Data  from  the  female  are  in  pink,  and  from  the  male  in  blue.  Times  are  in  seconds  and  angles  in 
degrees.  Maneuvers  indicate  a  bank  angle,  direction,  and  time  the  bank  was  held.  So  “30F,  1 
min”  means  a  30-degree  bank  was  flown  for  a  full  minute.  Results  indicate  inter-pilot,  as  well  as 
within  pilot  variability.  There  was  insufficient  time  to  fully  analyze  the  data,  but  none 
completely  contradicted  our  modeling  assumptions. 
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Table  2.  Flight  test  collated  data. 


Maneuver 

Description 

Times  from 

end  of 

maneuver  to 
perceived 
S&L 

Average 

Times 

Perceived 

max  bank 
angles 

Average 

max 

perceived 
bank  angle 

Perceived 

max 

illusory 
bank  angle 

Average 

max 

illusory 

bank 

angle 

30L,  1  min 

15.8;  21.6 

18.7 

1.4;  23.5 

12.5 

14.7;  0.0 

7.3 

30R,  1  min 

1.0;  6.2 

7.6 

OO 

iri 

3.7 

17.0;  21.6 

19.3 

45L,  1  min 

1.0;  33.8 

16.9 

25.1;  54.4 

39.7 

0.0;  0.0 

0.0 

45R,  1  min 

1.2;  9.4 

6.3 

1.9;  47.4 

29.1 

1.0;  11.2 

5.6 

30L  sub-threshold, 

1  min 

0.0;  44.8 

22.4 

0.0;  10.6 

5.3 

9.8;  18.3 

14.1 

30R  sub-threshold, 

1  min 

1.2;  28.4 

14.8 

0.0;  16.5 

8.3 

6.2;  7.3 

6.7 

30R,  10  sec;  30L, 

10  sec 

© 

oo 

2.0 

31.5;  42.2 

36.9 

0.0;  30.6 

15.3 

30L,  10  sec;  30R, 

10  sec 

17.0;  10.4 

13.7 

21.6;  32.4 

27.0 

28.7;  31.2 

29.9 

We  hope  to  have  time  during  follow-on  research  to  understand  these  data,  and,  even  better,  to  re¬ 
do  the  test  as  a  formal  experiment  with  fully  functioning  aircraft  state  recording  equipment.  In 
addition  to  comparing  actual  flight  data  to  subjective  perceptions,  we  would  like  to  flight  test  the 
efficacy  of  a  more  fully  developed  version  of  SO  AS. 


Aiding  Focused  Experiments 

The  first  aiding  focused  experiments  examined  the  usefulness  of  our  innovative  SD  Icon  on  a 
desktop  simulator.  The  second  examined  the  value  of  the  Icon  and  other  non-visual 
countermeasures  in  a  higher  fidelity  simulator  using  real  fighter  pilots,  rather  than  Academy 
cadets.  The  first  (Icon)  experiment  indicated  that  the  Icon  could  be  helpful  as  a  command 
display  to  help  pilots  recover  from  unusual  attitudes.  Cadet  volunteers  were  the  subjects. 

Readers  interested  in  details  should  see  the  first  International  Journal  of  Aviation  Psychology 
(IJAP)  manuscript,  which  has  been  accepted  for  publication,  in  Appendix  A. 

The  second  aiding  experiment  also  presented  subjects  with  sudden  unusual  attitudes.  They  flew 
recoveries  using  the  baseline  F-16  simulator  displays,  with  Icon  only  aiding,  and  with  Icon  plus 
voice  command  aiding,  or  Icon  plus  tactile  vest  command  aiding.  Results  proved  the  value  of 
aiding  over  the  baseline  as  there  were  significant  improvements  to  initial  response  times  and 
fewer  roll  reversal  errors.  In  the  worst  unusual  attitudes,  the  two  conditions  of  multisensory 
aiding  (Icon  plus  voice  and  Icon  plus  tactile)  were  significantly  better  than  the  two  other 
conditions  in  terms  of  initial  correct  response  times.  Again,  for  more  details  readers  should  see 
the  second  IJAP  draft  manuscript  (recently  submitted)  in  Appendix  A. 

We  used  traditionally  designed  human  factors  experiments  to  support  our  modeling  detection  and 
aiding  countermeasure  hypotheses.  The  culmination  of  our  dual  approach  to  SD  -  detection  and 
aiding  -  is  a  software  product  that  is  useful  for  post  hoc  accident  data  analyses  and  for  eventual 
use  as  an  in  situ  cockpit  aid.  The  software  is  explained  next. 
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3.4  Software  Design  &  Development 

Early  in  this  project  it  was  suggested  that  it  would  be  very  useful  to  have  a  software  tool  that 
allowed  analysts  and  investigators  to  evaluate  data  sets  from  actual  flights  with  the  SD  detection 
models.  At  the  time,  the  team  was  using  a  software  environment  that  supported  development 
and  testing  of  the  code  for  the  models.  While  the  focus  of  the  project  was  to  work  toward 
developing  a  cockpit  system,  creation  of  a  desktop  software  package  that  both  demonstrated  the 
algorithms  and  supported  actual  analysis  seemed  like  a  very  useful  intennediate  step.  The  result 
was  the  Spatial  Disorientation  Analysis  Tool  (SDAT). 

SDAT  is  now  a  Windows-based  desktop  software  package  complete  with  User  Guide  and  built- 
in  Help  documentation.  The  SDAT  User  Guide  describes  the  functioning  and  use  of  the  system 
in  detail  and  is  a  separate  deliverable  under  this  contract.  This  section  of  the  report  will  provide 
a  brief  summary  of  the  functionality  and  interface  in  terms  of  how  it  supports  the  design  and 
development  efforts  of  the  project.  In  addition,  this  section  briefly  describes  the  heuristic 
usability  analysis  that  was  performed  on  the  SDAT  interface,  and  the  patent  application  for  the 
system  in  general. 

Figure  21  shows  where  SDAT  fits  within  the  combined  SD  detection  and  pilot  aiding 
methodology.  The  tool  includes  all  the  SD  model  algorithms  for  perceived  attitude  prediction 
and  disorientation  illusion  sequences  as  described  in  Section  3.1  of  this  document.  In  addition,  it 
includes  a  demonstration  of  how  the  countenneasures  of  the  aiding  system  would  execute  to 
support  the  pilot  once  the  system  predicts  that  an  SD  illusion  is  likely  to  be  occurring.  Users  of 
the  software  can  load  flight  data  files  as  specifically  formatted  Excel  spreadsheets.  When  an 
analysis  of  the  file  is  executed,  both  the  detection  and  aiding  algorithms  evaluate  the  data  line  by 
line  and  provide  results  dynamically  in  real-time  or  faster.  While  the  analysis  tool  could  have 
been  designed  to  provide  a  single  set  of  results  following  tool  execution,  the  process  of  line  by 
line  analysis  and  results  more  closely  represents  how  the  system  would  function  within  a  cockpit. 
The  detection  system  would  evaluate  aircraft  motions  and  other  inputs  as  they  occur  and  the 
aiding  system  would  present  appropriate  and  perhaps  changing  levels  of  countermeasures  as 
necessary.  Many  of  the  set-up  and  execution  setting  options  in  the  tool  reflect  the  potential  use 
of  the  algorithms  as  part  of  a  dynamic  cockpit  system. 


Figure  21.  Detection  and  Aiding  as  implemented  in  SDAT. 


28 


SDAT  Detection  Analysis 

SDAT  extends  the  SD  detection  capability  into  the  realm  of  analysis  by  allowing  the  users  to 
modify  both  perception  prediction  and  illusion  model  values  and  providing  a  range  of  dynami¬ 
cally  displayed  results.  Figure  22  shows  the  initial  screen  display  when  SDAT  is  executed.  The 
interface  is  a  standard  Windows  design  with  pull  down  menu  options  and  a  toolbar.  The  main 
area  of  the  window  consists  of  a  two-layer  tabbed  interface  that  allows  the  user  to  set  initial 
conditions  for  analysis  and  view  various  results  options.  The  following  are  short  summaries  of 
each  of  the  options. 


Figure  22.  SDAT  initial  setup  screen. 

The  following  are  short  summaries  of  each  of  the  options  including  their  purpose  and  some  of 
the  reasoning  behind  their  functionality.  For  complete  details  on  the  use  of  the  tool,  see  the 
SDAT  User  Guide. 

Flight  Data  File  -  the  tool  allows  the  user  to  select  a  file  to  analyze  that  contains  the  flight  data. 
Files  must  be  in  Excel  with  appropriately  fonnatted  columns  of  flight  dynamics  data.  The  flight 
data  used  by  the  tool  could  also  be  available  to  an  SD  cockpit  detection  and  aiding  system  during 
flight. 

Illusion  Type  -  The  interface  lists  all  the  types  of  SD  illusions  currently  included  in  the 
detection  system.  The  General  Illusion  simply  looks  for  specified  deltas  in  actual  versus 
perceived  values  for  yaw  and  roll  rates  and  pitch  angles.  The  tool  allows  the  user  to  evaluate  a 
flight  data  set  for  each  individual  illusion  type  or  for  all  of  them  at  the  same  time.  The  ‘All’ 
option  more  closely  represents  how  the  detection  system  would  function  within  a  cockpit. 

Data  Rounding  and  Sampling  Rate  -  Some  of  the  flight  dynamics  data  sets  that  we  received 
(or  created  using  X-Plane,  a  desktop  flight  simulator)  have  sampling  rates  of  greater  than  8  Hz 
and  values  to  10  significant  digits.  For  the  purposes  of  modeling  human  perception,  it  is  not 


necessary  to  have  this  level  of  precision  within  the  data.  Changes  to  aircraft  attitude  values 
beyond  10ths  or  100ths  of  degrees  are  not  perceptible  by  humans.  Likewise,  variations  near  the 
1/10  of  a  second  range  can  relate  more  to  aircraft  vibration  than  actual  attitude  changes.  As 
such,  the  tool  allows  the  user  to  set  both  data  rounding  values  and  a  sampling  rate  that  better 
correspond  to  human  perceptions.  This  capability  filters  some  of  the  noise  in  the  attitude  data 
and  focuses  the  calculations  on  changes  significant  to  human  perception. 

Vestibular  Thresholds  -  As  described  in  the  section  on  SD  related  physiology  (Section  2.3) 
various  research  efforts  have  found  different  values  for  vestibular  detection  thresholds.  Indeed, 
thresholds  may  vary  across  individuals  or  circumstances.  The  tool  allows  the  user  to  set  specific 
threshold  values  for  use  in  the  semi-circular  canal  perception  model  for  both  roll  and  yaw. 

Washout  Function  -  The  tool  supports  both  linear  and  exponential  decay  functions  to  simulate 
the  effects  of  vestibular  washout.  Functional  coefficients  can  also  be  set  by  the  user  to  modify 
the  decay  rate  of  the  linear  function  and  the  shape  of  the  curve  for  the  exponential  function.  The 
default  value  of  0.12  for  the  exponential  function  reflects  the  rotation  perception  decay  results  of 
the  somatogyral  experiment  (Section  3.2). 

Sustained  Time  Window  -  This  setting  refers  to  the  Mulder’s  Law  implementation  (section  3.1) 
for  prolonged  acceleration  detection.  User  can  set  the  range  of  time  used  by  the  Mulder’s 
calculation  to  check  for  perception  of  combinations  of  otherwise  sub-threshold  instantaneous 
rotational  acceleration  values. 

In  addition  to  the  initial  set-up  window,  there  are  windows  for  the  individual  SD  illusions  that  we 
modeled:  Leans,  Graveyard  Spiral,  and  Coriolis.  Those  illusion  sequence  descriptions  are  next. 


Illusion  Sequence  Model  Settings 

Because  there  is  still  some  uncertainty  about  quantifying  misperceptions  and  recognizing  that 
there  is  variability  between  pilots  as  well  as  within  individual  pilots  under  varying  circum¬ 
stances,  the  tool  enables  the  user  to  alter  values  for  each  of  the  three  illusion  models  described  in 
Section  3.1  (and  the  General  Illusion)  to  vary  the  analytical  parameters.  Figure  2323  shows  the 
initial  settings  interface  for  the  Leans  model  (see  also  Figure  12).  The  basic  version  looks  for 
sub-threshold  roll  actions  starting  from  straight  and  level  flight  (Event  1).  The  user  might  choose 
to  search  for  a  Leans  sequence  that  initiates  from  a  greater  initial  roll  angle  or  rate  by  changing 
the  values  for  Event  1 .  In  addition,  the  level  of  certainty  can  be  varied  for  each  different  event  in 
the  sequence.  Likewise,  values  for  the  other  three  events  can  be  modified.  In  a  similar  way, 
settings  for  Coriolis,  Graveyard  Spiral  and  the  General  Illusions  can  be  changed.  In  this  manner, 
the  tool  supports  a  sensitivity  analysis  across  a  range  of  detection  settings. 

SDAT  outputs  include  a  whole  range  of  analysis  support  capabilities.  The  final  results  of  any 
analysis  include  time-stamped  detection  settings  and  values  in  tab-delimited  text  files.  These 
files  can  be  opened  for  further  analysis  or  presentation  in  any  software  package  that  supports 
data  in  spreadsheet  formats.  The  tool  also  uses  several  different  types  of  displays  to  present 
detection  results  dynamically  during  analysis  execution.  These  include  the  timing  of  the 
execution  for  each  event  in  each  illusion  sequence,  a  graphing  window  for  a  range  of  flight  data 
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and  results  values,  and  a  window  for  the  dynamic  rendering  of  actual  (blue)  versus  perceived 
(red)  aircraft  attitude  (Figure  24). 


Vestibular  }  Aircraft  State  Recovery  ;  Coriolis  ;  Graveyard  Spiral/  Leans  |  General 


Leans  Model  Inputs 

Value 

Default 

SD  Certainty 

Event  1  Roll: 

1.00 

1.00  deg 

Event  1  Roll  Velocity: 

1.00 

1.00  deg/sec 

|None 

3 

Event  2  Roll  Delta: 

5.00 

5.00  deg 

Event  2  Time: 

5.00 

5.00  sec 

|  N  o  1 1  e 

3 

Event  3  Vertical  Speed: 

-500.00 

-500.00  fpm 

|  Moderate 

3 

Event  4  Roll  Velocity: 

5.00 

5.0  deg/sec 

|  High 

3 

|  (1)  Sub-threshold  rod] 


r  =  roll  angle 
r'  =  roll  rate 
w  =  vertical  velocity 


(1)  From  a  user  defined  roll  angle  and 
“  q  velocity,  subthreshold  roll  accelerations 

(D  that  lead  to 

U  (2)  a  roll  change  of  >5  deg  for  at  least  5 
5  second  followed  by 

•S’  (3)  a  loss  of  altitude  (-w),  then 

(4)  an  above  threshold  roll  (faster  than  5 
(/)  deg/sec)  foil  ewe d  by 

O  (5)  reverse  head  angle,  of  r  >  5  deg 


Reset  to  Defaults  | 


Figure  23.  Leans  model  settings  screen. 
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Figure  24.  Example  SDAT  dynamic  analysis  screens. 


SDAT  Aiding  Demonstration 

In  addition  to  the  analysis  capabilities  of  the  detection  system,  SDAT  includes  the  algorithms 
developed  for  the  pilot  aiding  portion  of  the  system  (as  explained  in  Section  3.2).  During  an 


analysis,  the  tool  presents  the  changing  levels  of  countermeasures  that  the  aiding  system  would 
use  in  response  to  different  levels  of  disorientation  and  aircraft  risk  (i.e.,  consequence  severity). 
While  not  strictly  necessary  for  the  purpose  of  analyses,  this  capability  enables  demonstrating  the 
complete  SD  detection  and  pilot  aiding  system. 

Figure  25  shows  the  countermeasure  initiation  demonstration  screen.  The  left  side  shows  which 
illusion(s)  and  SD  certainty  levels  are  predicted  by  the  detection  system.  The  right  side  shows 
all  the  countermeasures  described  in  Section  3.2.  As  the  situation  changes  the  output  shows 
which  countermeasures  would  be  triggered  in  the  cockpit  to  aid  the  pilot.  This  screen  shot  shows 
that  under  the  current  situation  of  a  general  yaw  velocity  illusion,  the  cockpit  aiding  system 
would  be  using  the  icon,  audio  and  tactile  command  countermeasures  to  help  the  pilot  recover. 


Analysis  Setup  }  Illusion  Events  }  Results  Graphs  [  Animator3D  -  Countermeasures  | _ 

Spatial  Orientation  Aiding  System  (SPAS)  Countermeasures 

This  tab  is  used  to  demonstrate  the  countermeasures  portion  of  the  SOAS  cockpit  system.  It 
dynamically  shows  the  countermeasures  that  would  be  initiated  to  support  the  pilot  in  response  to 
spatial  disorientations  detected  in  this  flight  data  set. 


Predicted  Disorientation  Levels 

Resulting  Countermeasure  Actions 

Illusion  Type:  General  Yaw  Velocity 

Icon  Command: 

Yes 

Event  Flag:  One 

Audio  Cue: 

No 

Certainty  Level:  Moderate 

Tactile  Cue: 

No 

Audio  Commands: 

Roll  Left,  Pull  Up 

Tactile  Commands: 

Yes 

Olfactory  Cue: 

No 

Auto  Recovery: 

No 

Auto  Ejection: 

No 

Figure  25.  Countermeasure  initiation  demonstration  screen. 


In  the  same  way  that  the  tool  allows  users  to  modify  illusion  model  settings,  various  algorithms 
related  to  the  countermeasures  can  also  be  modified.  In  addition  to  demonstration,  this  allows 
for  the  testing  of  the  sequence  and  timing  of  countermeasure  initiation  during  flight  data 
sequences.  As  an  example  of  this  capability,  Figure  26  shows  a  screen  shot  of  recovery  code 
settings. 


Vestibular  |  Aircraft  State  -  Recovery  |  Coriolis  j  Graveyard  Spiral  !  Leans  General _ 

Recovery  Definition 

Following  an  SD  detection,  the  pilot  is  said  to  have  regained  control  of  the  AC  if  the  following 
conditions  have  been  met: 


AC  Pitch  angle  max  (+/-): 
AC  Roll  angle  max  (+/-): 
Yaw  Velocity  Delta  max: 


5.00 


5.00 


2.50 


Default 
5.00  deg 

5.00  deg 
2.50  deg/sec 


And  all  values  are  maintained  for  a 
minimum  of: 

Default 


Duration: 


2.50 


2.50  sec 


Figure  26.  Recovery  code  user  input  screen. 
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In  addition  to  the  Recovery  Definition,  the  tool  includes  options  for  modifying  algorithm  values 
for  the  Unusual  Attitude  Definition,  Time  to  Impact  Definition,  and  Perceived  Attitude  Angle 
Reset  Definition.  Each  of  these  algorithms  would  be  a  functional  part  of  any  cockpit  aiding 
system.  The  following  are  short  summaries  of  how  each  of  these  algorithms  interacts  with  both 
the  detection  and  aiding  systems. 

Recovery  Definition  -  Once  the  detection  system  has  determined  that  an  SD  situation  may  be 
occurring  and  countenneasures  have  been  activated,  the  system  needs  a  way  to  determine  if  the 
pilot  has  regained  control  of  the  aircraft  and  recovered.  The  definition  requires  that  the  aircraft 
pitch  and  roll  angles,  and  calculated  yaw  velocity  delta  are  within  specified  values  and  that  these 
values  are  maintained  for  a  specified  period  of  time.  Once  the  system  has  determined  that  the 
pilot  has  regained  control,  all  countenneasures  and  illusion  event  flags  are  turned  off. 

Unusual  Attitude  Definition  -  A  detected  SD  situation  may  not  necessarily  represent  a  loss  of 
aircraft  control.  Following  an  SD  detection,  if  an  unusual  attitude  is  also  detected,  the  system 
will  increase  the  level  of  countenneasure  intrusiveness  assuming  that  the  pilot  is  losing  control 
due  to  the  disorientation. 

Impact  Definition  -  The  impact  definition  is  used  by  the  aiding  system  to  determine  when  to 
initiate  either  the  auto-recovery  or  auto-eject  countenneasures.  Altering  the  time  to  impact 
values  allows  the  user  to  set  the  countenneasure  triggers  as  desired. 

Perceived  Attitude  Angle  Reset  Definition  -  The  purpose  of  this  algorithm  is  to  reset  the 
displayed  perceived  attitude  angles  of  pitch,  roll  and  yaw  to  the  actual  aircraft  values  in  the  3D 
Animator  window.  The  reset  does  not  affect  the  perceived  angular  velocities  or  any  other  values 
or  operations  within  the  detection  system.  The  reasoning  is  that  the  perceived  attitude 
calculations  are  limited  in  that  there  are  many  other  physiological  inputs  to  sense  actual 
orientation.  While  the  yaw  and  roll  rate  perceptions  are  well  represented,  the  resulting  attitude 
angles  are  not.  As  such,  that  algorithm  takes  advantage  of  long  periods  of  straight  and  level 
flight  in  a  data  set  and  low  perception  delta  values  to  reset  the  perceived  angle  values  being 
displayed. 


Heuristic  Usability  Analysis 

The  development  and  current  use  of  SDAT  has  been  primarily  by  the  same  group  of  people.  So, 
it  was  difficult  to  determine  what  sorts  of  usability  problems  the  software  design  might  contain. 
In  an  effort  to  increase  the  ease  of  use  and  understanding  of  the  tool  for  future  user  groups,  a 
heuristic  usability  analysis  was  performed  on  the  interface  as  part  of  our  design  and 
development. 

Heuristic  analysis  is  a  form  of  usability  evaluation  commonly  used  during  software  design.  The 
process  was  invented  by  a  Nielsen  (1994)  and  requires  the  use  of  interface  design  experts  rather 
than  users.  Sometimes  called  ‘discount  usability’  the  process  will  not  find  all  the  usability 
problems  but  is  less  expensive  than  fonnalized  usability  testing.  We  used  three  in-house  human 
factors  engineers  who  were  experienced  with  performing  heuristic  evaluations  but  who  had  no 
part  in  the  design  or  development  of  SDAT.  They  each  performed  independent  evaluations 
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using  a  list  of  common  interface  design  ‘rules  of  thumb’  (heuristics).  They  then  combined  their 
individual  analyses  into  a  single  report  that  prioritized  the  problems  they  found.  The  list  of 
heuristics  and  their  analytical  results  are  in  Appendix  B. 

The  results  of  the  analysis  proved  very  useful  to  the  design  team.  Due  to  timing  and  funding 
limitations,  though,  we  were  only  able  to  implement  a  few  of  the  changes  recommended  by  the 
evaluators.  We  focused  on  issues  that  were  listed  as  serious  and  were  also  able  to  implement 
some  of  the  quicker  changes.  A  more  complete  implementation  of  the  heuristic  team’s 
recommendations  will  have  to  wait  for  further  funding. 


Patent 

Because  of  the  innovative  nature  of  our  approach  and  its  embodiment  in  the  SDAT/SOAS 
software,  we  applied  for  a  US  patent.  We  filed  a  provisional  patent  application  (PPA)  on  May  6, 
2005  and  the  final  application  on  August  15  (Appendix  C).  We  have  yet  to  receive  feedback 
from  the  Patent  and  Trademark  Office  (PTO),  other  than  their  acknowledgment  that  they 
received  our  application  (Appendix  D). 

We  began  researching  the  feasibility  and  benefits  of  patenting  our  SD  tools  at  the  suggestion  of 
Bill  Ercoline,  with  the  subsequent  concurrence  of  our  technical  customers  and  their  contracting 
staff.  We  selected  the  Kansas  City-based  legal  firm  Lathrop  &  Gage,  L.C.  to  serve  as  counsel. 
Not  only  does  the  firm  has  an  established  and  highly  respected  intellectual  property  practice 
group,  but  a  number  of  its  most  experienced  attorneys  work  out  of  the  Boulder  office. 

We  were  motivated  to  file  a  PPA  before  publicly  disclosing  our  tools  at  the  Aerospace  Medical 
Association  conference  in  May  2005  where  we  had  an  exhibit.  Filing  a  PPA  accomplished 
several  things.  First,  it  provided  us  with  an  official  filing  date  that  allowed  us  to  publicly 
disclose  the  invention  without  jeopardizing  our  domestic  or  foreign  filing  rights.  This  is 
especially  important  for  foreign  filing,  since  most  countries  do  not  offer  a  one-year  grace  period 
for  filing  after  public  disclosure  as  does  the  US.  It  also  prevented  our  own  public  disclosures 
from  being  used  as  “prior  art”  against  us.  Finally,  it  placed  the  burden  on  potential  competitors 
to  demonstrate  proof  of  concept  prior  to  our  filing  date. 

Our  strategy  for  the  PPA  (and  consequently  the  regular  patent  application)  was  to  claim  the 
broadest  range  of  concepts  and  methodologies  possible.  We  began  by  widening  the  scope  of  our 
invention  to  support  SD  countenneasures  and  analysis  in  any  context,  not  just  aviation.  With 
this  thinking,  our  tool  could  conceivably  be  used  to  assist  disoriented  divers,  firefighters, 
astronauts,  or  anyone  else  who  is  regularly  exposed  to  conditions  conducive  to  vestibular  SD 
illusions.  Additionally,  we  avoided  mention  of  specific  platforms  and  computer  programs  used 
to  build  our  tool.  A  generalized  description  of  how  the  tool  was  constructed  lays  claim  to  all  its 
possible  manifestations  and  thereby  prevents  competitors  from  building  essentially  the  same 
device  with  insignificant  differences. 

Due  to  funding  limitations,  we  elected  to  not  file  a  Patent  Cooperation  Treaty  (PCT)  application 
to  initiate  the  foreign  filing  process;  foreign  patent  applications  are  very  expensive.  A  PCT 
application  is  not  a  patent  application.  Rather,  it  is  a  form  of  international  registration  for  an 


invention  that  allows  the  inventor(s)  to  research  the  feasibility  and  benefits  of  obtaining  patent 
rights  in  foreign  countries  before  making  a  significant  commitment  of  time  and  money.  The  US 
application  cost  $15,000;  each  foreign  filing  costs  about  half  of  that  of  the  US  filing,  not 
counting  language  translation  services. 


3.5  Accident  Data  Analyses 

During  the  early  development  of  both  the  detection  and  aiding  algorithms,  we  used  flight  data 
sets  generated  from  a  desktop  flight  simulator  (X-Plane).  The  scenarios  were  flown  by  project 
personnel  to  carefully  recreate  the  specific  illusions  and  sequences  we  were  trying  to  detect.  The 
sequences  were  often  followed  by  periods  of  erratic  movement  to  simulate  a  loss  of  control. 
Finally,  we  often  flew  recoveries  to  simulate  a  pilot’s  response  to  supporting  countermeasures. 
These  data  sets  proved  very  useful.  The  flight  simulator  could  output  all  the  flight  dynamics  data 
we  needed  and  the  sequences  allowed  us  to  debug  our  detection  and  aiding  algorithms  as  coded 
in  SDAT.  However,  since  the  sequences  were  developed  specifically  to  work  with  our  models, 
we  were  uncertain  how  well  the  algorithms  would  perform  with  data  sets  from  real  flights. 

We  were  very  fortunate  to  obtain  12  such  data  sets  from  actual  SD  mishaps.  The  Air  Force 
Safety  Center  (AFSC),  Navy  Aeromedical  Research  Lab  (NAMRL),  and  National  Transportation 
Safety  Board  (NTSB)  were  all  extremely  supportive  of  our  efforts  and  were  gracious  enough  to 
share  not  only  flight  data  sets  from  investigated  mishaps  but  also  the  associated  safety  board 
investigatory  findings.  These  data  sets  and  the  ability  to  evaluate  the  conclusions  from  SDAT 
proved  invaluable  to  this  project.  We  are  indebted  to  our  colleagues  from  each  of  these 
organizations  and  look  forward  to  continuing  to  work  with  each  of  them. 

Each  of  these  data  sets  was  from  an  actual  SD  incident  that  had  been  previously  investigated.  It 
was  agreed  that  our  use  of  the  data  was  for  our  own  development  purposes  and  that  none  of  our 
results  would  be  used  to  modify  the  conclusions  from  any  of  the  investigations.  In  addition,  the 
use  of  the  AFSC  data  sets  required  us  to  sign  a  non-disclosure  agreement;  so,  there  will  be  no 
discussion  of  those  data  sets  in  this  report.  In  addition  to  aiding  us  with  our  system  development, 
we  created  analysis  reports  for  each  data  set  containing  the  results  from  SDAT.  These  reports 
were  provided  to  each  organization  who  gave  us  data  sets.  The  remainder  of  this  section 
includes  summaries  of  four  of  these  analyses.  Unless  otherwise  controlled,  the  complete  text  of 
these  analyses  is  available  upon  request.  It  should  also  be  noted  that  none  of  these  data  sets 
included  head  position  values  for  the  pilot.  As  such,  the  Coriolis  model  was  not  used  as  part  of 
any  of  these  analyses. 

In  addition  to  receiving  actual  flight  data  sets,  we  created  one  flight  data  set  from  a  video 
recording  of  a  HUD  during  an  apparent  disorientation.  The  following  SDAT  summaries  include 
two  data  sets  from  the  NTSB,  one  from  the  Navy,  and  the  recreated  HUD  tape  sequence.  In  each 
case,  the  basic  event  sequence  is  described  and  the  main  SDAT  findings  are  presented  along  with 
a  comparison  of  the  conclusions  from  each  investigation  for  the  NTSB  and  Navy  data  sets.  Also, 
we  discuss  how  the  analysis  of  each  data  set  supported  our  development  process.  We  conclude 
with  our  detection  score  for  all  12  data  sets.  Each  SDAT  analysis  was  initially  perfonned 
independent  of  the  official  findings  in  order  to  be  a  “blind”  test  of  SDAT.  Following  each 
analysis  we  compared  the  SDAT  results  with  the  conclusions  of  the  incident  investigators. 
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Consequently,  these  comparisons  function  as  a  basic  validation  of  the  SD  detection  capabilities 
of  our  system. 

NTSB  Data  Set  1:  Strasburg 

During  a  night  flight  in  bad  weather,  this  King  Air  200  lost  electrical  power.  Just  over  2  minutes 
later  during  a  long  right  turn,  it  spiraled  into  the  ground  killing  all  aboard  (NTSB,  2003).  Figure 
27  shows  graphs  of  aircraft  altitude,  roll  angle,  yaw  rate  and  perceived  yaw  rate.  SDAT  detected 
a  Leans  illusion  sequence  with  the  sub-threshold  roll  resulting  in  over  10  deg  of  roll  change 
starting  around  time  90  and  extending  for  approximately  20  seconds.  SDAT  also  detected  a  yaw 
rotation  washout  with  an  increasing  perception  delta  reaching  at  least  10  deg/sec  followed  by  the 
apparent  loss  of  control. 


Figure  27.  SDAT  Leans  &  Washout  detections  with  graphs  of  altitude,  roll  angle  and  yaw  rate. 


The  comparison  with  the  NTSB  results  showed  that  the  detected  Leans  sequence  started  just  after 
the  loss  of  electrical  power  (Figure  28).  The  yaw  washout  detection  coincided  with  the 
increasing  final  spiral.  The  NTSB  concluded  that  a  Graveyard  Spiral  illusion  contributed  to  this 
fatal  accident. 
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Figure  28.  NTSB  ground  track  and  events  timeline  combined  with  SDAT  findings. 


The  model  of  the  Graveyard  Spiral  illusion  includes  three  turning  events  rather  than  the  single 
one  that  occurred  here.  While  the  current  detection  system  calculates  the  resulting  yaw  rate 
perception  delta,  future  versions  of  the  Graveyard  Spiral  model  may  need  to  include  a  sub¬ 
threshold  beginning  to  an  increasing  yaw  washout  event. 


NTSB  Data  Set  2:  Hillsboro 

During  a  night  flight  in  bad  weather,  the  airplane  carrying  Governor  Carnahan  lost  the  primary 
attitude  indicator  shortly  after  takeoff.  After  nearly  10  minutes  of  working  with  an  air  traffic 
controller  (ATC)  and  reporting  attitude  difficulties,  the  pilot  apparently  lost  control  and  crashed 
(NTSB,  2002).  Figure  29  shows  the  actual  yaw  rate  and  the  calculated  perceived  yaw  rate. 
SDAT  detected  two  instances  of  yaw  rotation  washout  resulting  in  perception  deltas  of  over  3 
deg/sec,  the  second  of  which  occurred  just  prior  to  the  apparent  loss  of  control. 
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Figure  29.  Actual  and  perceived  yaw  rate  with  detected  washout  events. 
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Figure  30  shows  the  ground  track  and  communication  timeline  of  the  flight.  In  addition  to  the 
two  instances  of  washout,  SDAT  detected  three  Leans  sequences.  Each  are  associated  with  the 
three  major  turns  taken  during  the  last  half  of  the  flight  sequence.  The  comparison  with  the 
NTSB  findings  showed  that  the  first  and  third  turns  were  directed  by  ATC.  As  such,  it  is  un¬ 
likely  that  the  two  Leans  sequences  associated  with  these  turns  were  an  issue  as  they  were  inten¬ 
tionally  initiated.  However,  the  second  Leans  sequence  coincides  with  a  turn  that  apparently  was 
not  directed  by  ATC  and  may  have  been  due  to  SD.  Finally,  the  NTSB  concluded  that  a  Coriolis 
illusion  may  have  contributed  to  the  final  apparent  loss  of  control  as  the  pilot  may  have  been 
trying  to  look  across  the  cockpit  to  use  the  secondary  attitude  indicator.  The  washout  detected 
during  the  last  turn  is  consistent  with  an  initiating  event  of  the  Coriolis  illusion. 


Ground  Track  Relative  to  St. Louis  ASR  Antenna 
with  Selected  ATC  Transmissions 
N8354N  10/16/00 
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Figure  30.  NTSB  ground  track  and  event  timeline  with  SDAT  detections. 


Navy  Data  Set 

This  data  set  was  received  with  almost  no  background  information.  Based  on  the  actual  yaw  rate 
graph  in  Figure  31,  the  aircraft  was  in  a  long  left  turn,  briefly  stopped  turning,  and  then 
continued  back  into  the  left  turn  with  a  heading  change  rate  of  up  to  5  deg/sec.  SDAT  did  not 
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detect  any  Leans  or  Graveyard  Spiral  illusion  sequences.  However,  it  calculated  that  nearly  all 
the  turn  accelerations  were  below  the  vestibular  threshold,  so  that  the  pilot  would  sense  almost 
none  of  the  turn  as  shown  by  the  red  line  of  the  graph. 


Continuing  our  analysis,  SDAT  calculated  a  steadily  increasing  pitch  angle  perception  delta  near 
the  end  of  the  data  set.  Figure  32  shows  altitude,  airspeed,  and  pitch  angle  for  the  flight  as  well 
as  perceived  pitch  angle  and  the  two  periods  of  pitch  angle  perception  delta.  As  airspeed 
increases  while  climbing,  starting  from  time  125,  the  perceived  pitch  increases.  The  pilot  either 
does  not  feel  the  actual  decrease  in  pitch  over  the  next  20  seconds  or  is  intentionally  responding 
to  the  perception  of  increasing  pitch  up  by  pushing  the  nose  down.  The  process  accelerates  as 
airspeed  increases  due  to  the  loss  in  altitude  starting  at  about  time  165.  These  findings  coincide 
with  the  Navy’s  conclusion  of  a  pitch  perception  problem. 


Figure  32.  Altitude,  airspeed  and  pitch  angle  graphs  with  pitch  perception  deltas. 
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The  analysis  of  this  data  set  was  useful  in  two  ways.  First,  it  confirmed  that  our  developing 
analysis  process  was  effective.  Although  we  were  initially  focused  on  the  sub-threshold  yaw 
accelerations,  the  completion  of  the  detection  process  allowed  us  to  find  the  pitch  perception 
problem  related  directly  to  a  somatogravic  illusion.  Second,  it  confirmed  that  our  pitch 
perception  model  was  functioning  well,  but  highlighted  the  need  for  a  pitch-based  illusion  model 
as  part  of  the  SD  detection  system. 


HUD  Recording  Data  Set 

The  recording  of  this  FIUD  tape  was  made  available  to  us  with  no  information  about  the  context 
of  the  flight  other  than  it  occurred  at  night  and  might  have  included  the  use  of  night  vision 
goggles.  From  an  SD  standpoint,  it  is  compelling  because  the  audio  content  shows  that  the  pilot 
is  aware  of  his  disorientation  and  is  eventually  able  to  recover.  We  used  video  editing  software 
to  slice  the  first  70  seconds  into  1  Hz  segments.  We  obtained  most  of  the  necessary  flight 
dynamics  data  from  values  on  the  HUD  (Figure  33).  For  pitch  and  roll  we  used  a  graphics 
program  to  measure  angles  relative  to  a  horizontal  reference  line  inserted  into  each  image  (Figure 
34). 
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Figure  33.  Flight  data  value  locations  on  the  HUD  tape  recording. 
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Figure  34.  Technique  of  obtaining  pitch  and  roll  angles  from  the  HUD  tape. 
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Figure  35  shows  the  graph  of  the  dramatic  rolling  action  that  occurred  during  the  flight  sequence. 
Included  is  a  subset  of  the  audio  as  transcribed  from  the  recording.  F2  is  flying  in  the  wing 
position  relative  to  lead  aircraft  FI.  FI  requests  approach  vectors  to  split  F2  from  the  fonnation 
in  preparation  for  landing  (our  assumption  from  the  audio  and  our  flight  experiences).  During 
this  communication,  F2  begins  to  experience  increasing  roll  oscillations.  His  first  expletive  at 
time  25  seconds  seems  to  indicate  that  he  is  aware  of  some  sort  of  problem.  Following  an  initial 
roll  inversion,  he  calls  “Lost  wingman.”  and  follows  with  another  expletive  as  the  roll  inversions 
continue.  He  responds  to  a  position  request  from  the  flight  leader  indicating  his  awareness  of  his 
disorientation.  Approximately  six  seconds  after  the  flight  lead  helps  to  focus  the  F2  pilot  on  the 
primary  attitude  indicator,  he  regains  control  of  the  aircraft  and  seems  to  proceed  without  further 
incident. 


Given  the  apparent  roll  control  problems,  we  expected  to  find  some  sort  of  Leans  illusion  issue 
somewhere  between  5  and  25  seconds  into  the  flight  sequence.  SDAT,  however,  did  not  find 
illusion  sequences  of  any  kind;  it  found  that  most  of  the  rolling  actions  were  supra-threshold. 
Indeed,  as  the  roll  rate  data  for  the  first  25  seconds  of  the  sequence  shows  (Figure  36),  the  only 
sub-threshold  change  occurs  for  about  three  seconds  starting  at  time  19  seconds.  While  the  result 
is  a  roll  rate  perception  delta  of  about  3  to  4  deg/sec,  it  seems  very  unlikely  that  such  a  relatively 
small  value  alone  could  account  for  the  subsequent  events  —  the  multiple  roll  inversions. 
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Figure  36.  Actual  and  perceived  roll  rate  and  small  perception  delta. 


A  presentation  and  discussion  of  these  findings  surprised  a  number  of  other  SD  researchers 
familiar  with  this  HUD  tape  sequence  and  indicated  to  them  that  the  disorientation  and  apparent 
control  problems  may  have  resulted  from  something  other  than  a  vestibular-based  perception 
issue  alone,  such  as  visual  motion  confusion  of  the  inside-out  representation  of  attitude  on  the 
HUD  (Previc  and  Ercoline,  1999).  Our  use  of  this  data  set  demonstrates  the  capability  of 
generating  flight  data  from  a  source  other  than  a  flight  data  recorder.  It  is  possible,  though,  that 
our  process  for  creating  the  data  set  may  have  introduced  small  errors  to  the  angular  rotation 
values.  In  any  case,  it  demonstrates  how  a  tool  such  as  SDAT  can  be  useful  during  an  overall 
analysis  of  a  flight  sequence  incident.  The  recording  also  demonstrates  how  an  audio  command, 
as  performed  by  the  flight  lead  in  this  case,  can  function  as  a  very  effective  countermeasure. 


Analysis  Comparison  Summary 

Figure  37  shows  the  summary  score  of  SDAT  results  as  compared  with  the  investigation  results 
for  each  of  the  12  data  sets  we  analyzed  during  this  project.  The  two  NTSB  and  one  Navy  data 
set  were  each  good  matches.  The  HUD  tape  data  set  was  somewhat  inconclusive  and  is  hard  to 
evaluate  without  additional  information.  Of  the  8  data  sets  provided  by  AFSC,  we  had  a  good 
positive  match  for  one  of  them.  For  two  of  them  we  had  good  negative  matches;  that  is,  SDAT 
did  not  find  any  vestibular  SD  problems  and  the  investigators  concluded  that  visual  illusions,  not 
vestibular  illusions,  contributed  to  the  mishaps.  In  the  two  cases  where  SDAT  results  did  not 
match  the  investigators  findings  at  all,  further  information  showed  that  the  use  of  our  vestibular- 
based  SD  illusion  models  was  inappropriate  because  of  circumstances  that  point  to  non- 
vestibular  illusions.  In  another  two  cases,  the  comparison  is  inconclusive.  While  both  the 
investigators  and  SDAT  indicate  there  were  SD  issues,  there  is  no  clear  match.  Finally,  we  were 
unable  to  complete  the  analysis  of  one  of  the  data  sets  as  it  appeared  to  be  corrupted  (marked 
“N/A”  in  Figure  37). 
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Air  Force:  8  data  sets  NTSB:  2  data  sets  Navy:  1  data  set 


Figure  37.  SD  flight  data  analysis  comparison  results. 


Based  on  these  results,  we  conclude  that  our  SD  detection  system  in  the  form  of  an  analysis  tool 
is  effective  at  finding  vestibular-based  perception  problems  in  real  flight  data. 
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4  Recommendations  for  Future  Research 

As  with  most  projects,  the  momentum  for  the  accomplishments  we  would  like  to  achieve 
exceeded  the  time  and  funding  that  we  had  to  complete  them.  With  this  in  mind,  we  have 
surveyed  the  items  that  we  were  either  unable  to  complete  or  should  be  considered  for  future 
research  and  design  efforts.  The  list  is  divided  into  the  following  subject  areas: 

•  Improvements  to  the  Detection  System 

•  Improvements  to  the  Aiding  System 

•  SOAS  Detection  and  Aiding  System  Testing  and  Integration 

•  Spatial  Disorientation  Research 

•  SDAT  Software  Enhancements 

•  Applications  to  Other  Topic  Areas 

Each  focus  area  and  topic  includes  a  short  description  of  the  potential  future  work  and  the 
reasons  we  feel  it  is  needed.  This  list  can  be  used  for  future  proposals  as  additional  funding  and 
support  becomes  available. 

4.1  Improvements  to  the  Detection  System 

The  SD  detection  system  has  proven  its  ability  to  detect  a  number  of  SD  situations.  However, 
there  are  a  number  of  improvements  that  can  be  done  to  increase  both  the  depth  and  breadth  of 
detection  capabilities.  If  the  system  is  eventually  to  function  within  an  integrated  cockpit 
environment,  these  types  of  enhancements  should  be  considered  a  required  step. 

Integration  of  Perception  Calculations 

The  calculations  for  the  perception  of  angular  rotation  and  angle  perception  used  in  the  current 
version  of  the  detection  system  are  functionally  isolated  from  each  other.  They  have  proven 
effective  in  predicting  portions  of  attitude  perception.  However,  our  vestibular  system  integrates 
orientation  inputs  from  several  sources  at  once  to  calculate  overall  attitude  perception.  We  will 
need  to  determine  how  important  each  additional  input  is  and  if  it  affects  the  SD  detection 
capability.  Two  specific  cases  are  likely  to  be  important  future  enhancements: 

•  The  perception  of  bank  angle  can  be  represented  by  an  integration  of  roll  rate  perception 
and  applied  G  forces  as  they  affect  the  proprioceptive  system. 

•  In  addition  to  the  loss  of  the  sense  of  rotation  caused  by  washout  during  a  continuous 
turn,  the  pilot  may  also  lose  the  feeling  of  the  hank  angle  associated  with  the  turn.  The 
result  is  the  feeling  of  straight-and-level  flight  while  still  banked  and  turning.  Combining 
the  rotational  washout  calculation  with  the  bank  angle  perception  calculation  will  be 
needed  to  accurately  reflect  this  sequence. 

Extension  of  Current  Illusion  Models 

The  current  version  of  the  detection  system  includes  illusion  models  for  Leans,  Graveyard  and 
Coriolis.  The  models  have  proven  effective  at  detecting  these  illusions  in  actual  flight  data  sets 
but  only  under  analysis  conditions  when  model  values  can  be  modified  and  flight  data  examined 
closely.  In  several  cases,  the  models  helped  to  direct  the  focus  of  the  analyst  to  SD  sequences 
that  were  similar  to  the  models  but  represented  variations.  The  most  dramatic  example  is  the 
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graveyard  spiral  sequence  that  begins  with  sub-threshold  roll  and  yaw  movements  (as  exhibited 
in  the  Strasburg  incident)  rather  than  a  sequence  of  rotational  washout,  illusory  rotation,  and 
erroneous  counteraction.  Future  versions  of  the  detection  system  should  include  additional 
versions  or  extensions  to  the  current  illusion  models.  The  leans  and  graveyard  spiral  models 
would  also  benefit  from  emerging  data  regarding  the  integration  of  roll  and  yaw  rate  information 
as  mentioned  above. 

Somatogravic  Illusion 

While  the  current  physiological  portions  of  the  SD  detection  system  include  a  calculation  of 
perceived  pitch  angle,  these  values  are  not  used  within  an  illusion  model  sequence.  The 
somatogravic  illusion  is  a  common  occurrence  resulting  from  the  misperception  of  pitch  angle. 
Further  development  of  the  detection  system  should  include  the  development  of  a  somatogravic 
illusion  model.  Similar  to  the  model  of  the  graveyard  spiral  sequence,  the  somatogravic  model 
would  start  with  initiating  events  that  detect  pitch  perception  deltas  of  specified  amounts  and 
lasting  for  specified  time  periods.  The  model  might  be  designed  to  detect  increasing  pitch 
perception  deltas  as  occurs  in  the  inversion  illusion.  It  could  also  include  events  for  apparent 
loss  of  aircraft  control,  such  as  high  negative  vertical  velocities  when  pitched  down,  or  aircraft 
dynamics  indicative  of  a  stall  situation  when  pitched  up. 

Visual  Illusions 

The  current  detection  system  is  focused  on  vestibular-based  illusions.  However,  there  are  a  host 
of  visual  illusions  that  seriously  effect  pilots  during  flight.  While  it  is  unknown  exactly  how 
such  illusions  could  be  modeled,  a  research  and  development  process  focused  on  this  goal  could 
dramatically  increase  the  applicability  of  an  SD  detection  system,  and  we  have  some  hunches 
based  upon  our  experiences.  As  an  example,  the  HUD  tape  analysis  would  have  benefited  from 
a  visual  illusion  component. 

Evaluation  of  Pilot  Position  Assumption 

A  number  of  simplifying  assumptions  were  made  during  the  development  of  the  initial  vestibular 
perception  calculations.  In  each  case,  the  purpose  of  the  assumptions  was  to  simplify  the 
modeling  process.  While  the  ability  of  the  models  to  detect  SD  under  analysis  conditions  has 
been  demonstrated,  future  versions  of  the  detection  system  may  need  to  use  higher  fidelity 
perception  calculations.  This  may  be  especially  true  when  the  system  is  integrated  with  actual 
cockpit  systems.  The  following  is  a  list  of  these  assumptions  as  focus  areas  for  future  efforts: 

•  We  assumed  that  the  pilot  was  positioned  at  the  aircraft  center  of  gravity  (CG)  such  that 
aircraft  angular  and  linear  accelerations  could  be  applied  directly  to  the  vestibular 
system.  Since  the  pilot  is  typically  not  located  at  the  CG,  we  need  to  detennine  if  the 
resulting  changes  to  vestibular  inputs  are  large  enough  to  affect  the  accuracy  of  the 
detection  system.  This  is  especially  problematic  as  the  effect  may  be  different  for  each 
different  type  of  aircraft. 

•  We  used  linear  approximations  to  calculate  the  angular  velocity  and  acceleration  values. 
We  need  to  determine  if  representing  more  accurate  aircraft  aerodynamics  affects  the 
accuracy  of  the  detection  system.  This  is  especially  problematic  as  the  effect  may  be 
different  for  each  different  type  of  aircraft.  For  example,  linear  approximations  may  be 
acceptable  for  a  small  single-engine  Cessna,  but  inadequate  for  an  F-16. 
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•  We  assumed  that  the  pilot  kept  his  head  position  in  line  with  the  aircraft.  Small 
movements  of  the  head  will  cause  the  actual  inputs  to  the  vestibular  system  to  differ 
from  the  motions  of  the  aircraft.  We  need  to  detennine  if  head  motions  represent  a  large 
enough  effect  on  perception  or  if  they  can  be  considered  secondary  to  the  larger  inputs 
from  the  aircraft  motion.  In  cases  where  we  have  head  position  data,  we  should  use 
them.  If  the  data  set  does  not  have  head  position  data,  then  we  cannot  use  them  and  will 
continue  to  use  simplifying  assumptions. 

4.2  Improvements  to  the  Aiding  System 

The  combination  of  pilot  workload,  aircraft  state,  and  countermeasure  presentation  developed 
during  this  project  has  proven  to  be  effective  in  aiding  time  to  recovery  from  unusual  attitudes. 
Enhancements  to  this  system  may  further  increase  its  effectiveness.  The  following  are 
descriptions  of  some  of  the  ideas  for  future  research  and  development  in  this  area. 

Workload  &  Stressor  Assessments 

To  further  enhance  SOAS,  we  need  to  calibrate  our  SAVVOY  workload  scores  to  more 
accurately  assess  pilot  workload  in  all  of  the  sensory  channels  in  which  we  are  interested.  This 
should  include  the  effects  of  stressors,  such  as  fatigue,  combat  for  a  new  pilot,  “go  pills,”  etc. 
There  are  indications  in  the  research  that  the  effects  of  these  types  of  stressors  on  vestibular 
physiology  can  alter  how  spatial  disorientations  occur.  Alion/MA&D  has  a  great  deal  of 
experience  with  accounting  for  human  stressors  in  military  systems,  and  this  expertise  could  be 
of  great  value  in  determining  the  effects  such  stressors  would  have  on  pilot  workload. 

Countermeasure  Habituation  and  Sensory  Fatigue 

Another  necessary  enhancement  to  the  application  of  countermeasures  is  to  more  accurately 
account  for  habituation  and  sensory  fatigue  to  determine  timing  for  the  various  countermeasures. 
Using  human  physiology  infonnation,  we  could  ensure  that  cues  are  less  likely  to  be  ignored  due 
to  habituation  and  sensory  fatigue.  For  example,  in  Phase  I,  we  learned  that  tactile  cues  should 
be  used  for  a  maximum  of  10-20  seconds  due  to  the  likelihood  of  habituation. 

4.3  SOAS  Detection  and  Aiding  System  Testing  and  Integration 

While  SOAS  performed  well  during  the  testing  and  experiments  performed  during  this  project, 
there  is  more  that  needs  to  be  done  to  fully  integrate  the  detection  and  aiding  components.  The 
working  relationship  developed  with  USAFA  during  the  course  of  this  project  could  be  a  key 
element  in  continuing  these  efforts.  A  large  multi-year  research  plan  could  be  generated  if  their 
facilities  can  be  maintained  and  if  a  USAFA  counterpart  is  available  to  oversee  the  research  and 
cadet  involvement.  The  following  are  suggestions  for  additional  integration  and  testing  that 
needs  to  be  done  in  collaboration  with  USAFA  and  others. 

SOAS  Evaluation 

While  we  were  able  to  capture  flight  test  data  to  further  calibrate  and  validate  our  models,  we  did 
not  conduct  a  full  flight  test  experiment  of  SOAS  due  to  resource  constraints.  Although,  our 
models  functioned  well  within  the  analysis  environment,  a  set  of  flight  test  experiments  is 
necessary  to  validate  SOAS.  One  of  the  flight  tests,  for  example,  should  explore  the  precise 
timing  relationship  between  visual,  voice,  and  tactile  cues  and  commands. 
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Simulator  Integration 

During  the  course  of  this  project,  we  started  the  process  of  integrating  the  countermeasure 
presentation  sequence  with  a  flight  simulation  system  in  preparation  for  an  experiment  designed 
to  test  the  efficacy  of  SOAS.  Simulator  availability  issues  prevented  us  from  completing  this 
effort.  Future  work  should  involve  integrating  SOAS  with  a  motion-based  simulator,  a  higher 
fidelity  stationary  simulator,  and/or  a  centrifuge-based  simulator  as  part  of  further  development 
and  testing. 

Helicopter  Data  with  Head  Position 

We  used  real  flight  data  sets  from  the  AF,  Navy  and  NTSB  but  did  not  obtain  any  from  the 
Army.  Specifically  we  would  like  to  evaluate  the  detection  system  with  SD  events  that  occurred 
in  helicopter  flights.  Such  events  are  of  interest  because  of  the  unique  rotary  dynamics  and  the 
potential  availability  of  head  position  data  through  the  head  mounted  targeting  systems. 

Detection  System  False  Positive  Evaluation 

While  we  obtained  and  used  real  flight  data  sets  from  SD  incidents,  we  did  not  obtain  any  that 
represented  combat  maneuvers  without  the  occurrence  of  SD  (e.g.,  Red  Flag  data).  Even  though 
SDAT  detected  SD  incidents,  it  needs  to  be  evaluated  for  false  positives.  This  will  require  the 
use  of  additional  flight  data  sets  and  perhaps  model  modifications. 

SD  Detection  Certainty  and  Visibility 

The  most  likely  false  SD  detections  for  a  cockpit  system  will  be  when  the  pilot  is  in  a  VFR 
environment.  Visual  dominance  clearly  impacts  SD.  If  SDAT/SOAS  could  determine  that  the 
pilot  is  visual  and  attending  to  his/her  orientation,  that  would  greatly  improve  our  SD  certainty 
level  assessment.  At  the  AsMA  conference  in  May  2006,  Art  Estrada  of  the  US  Army 
Aeromedical  Research  Lab  at  Ft  Rucker  presented  the  initial  successes  of  an  airborne  visibility 
indicator.  Such  a  system  could  provide  data  to  SOAS/SDAT  in  a  fully  integrated  future  cockpit. 

Transition  to  an  Actual  Cockpit  System 

In  addition  to  all  the  testing  and  integration  efforts  already  presented,  more  is  needed  in  order  for 
SOAS  to  be  ready  for  an  actual  integrated  cockpit  system.  Certainly  a  thorough  process  of  code 
verification  as  well  as  further  validation  testing  is  needed.  In  addition,  the  process  of  integrating 
it  within  the  cockpit  software  environment  will  likely  require  working  with  a  flight  systems 
development  company.  Finally,  there  will  be  the  FAA’s  process  of  certification  as  a  flight  ready 
system. 

4.4  Spatial  Disorientation  Research 

There  is  much  to  do  to  more  fully  understand  both  the  depth  and  breadth  of  the  SD  problem.  The 
following  is  a  list  of  potential  research  efforts  that  would  provide  additional  information  and 
understanding  of  SD. 

Mission  Impact  Evaluation 

Disorientations  that  do  not  result  in  the  loss  of  the  aircraft  or  crew  may  still  have  effects  on 
mission  completion.  For  instance,  the  time  lost  while  regaining  orientation  may  negatively 
impact  mission  time  lines.  A  thorough  evaluation  of  these  effects  may  show  that  SD  is  an  even 
greater  problem  than  currently  believed.  Efforts  could  include: 
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•  Poll  all  active  military  pilots  for  their  personal  SD  stories  and  how  they  survived,  plus 
asking  about  the  mission  impact  of  the  SD  event. 

•  Analyze  all  military  and  NTSB  accident  data  sets  looking  for  any  SD  link.  SD  has  been 
under-appreciated  particularly  in  fonnal  GA  accident  investigations,  and  there  is  much 
we  could  do  if  that  link  were  more  fully  articulated.  The  Aviation  Safety  Reporting 
System  (ASRS)  database  should  be  examined  for  SD  incidents,  as  well. 

Foreign  Country  SD  Comparison 

In  the  US,  SD  tends  to  be  underreported  because  of  a  stigma  associated  with  its  occurrence.  It 
would  be  useful  to  understand  how  pilots  and  safety  personnel  in  other  countries  view  the 
problem.  If  it  is  viewed  as  a  more  socially  neutral  medical  condition  in  other  places,  there  may 
be  a  way  such  information  could  be  used  to  change  negative  US  attitudes  and  thereby  promote 
safety.  Such  an  educational  effort  could  include  dramatic  presentations  of  SD  (such  as  the  HUD 
tape)  to  begin  frank  discussions  of  the  problem  with  safety  personnel  who  will  protect  pilot 
confidentiality. 

4.5  SDAT  Software  Enhancements 

The  current  version  of  the  SDAT  software  functions  well.  However,  new  features  and  additional 
ideas  are  always  part  of  the  software  development  process.  As  part  of  further  tool  development 
we  would  like  to  work  with  personnel  from  flight  safety  and  accident  investigation  teams  to 
better  understand  their  practices  and  needs.  Such  efforts  will  likely  indicate  changes  or 
refinements  to  the  tool  that  will  allow  it  to  better  support  those  needs.  The  following  is  a  list  of 
other  features  and  functions  that  could  be  added  in  future  versions  of  SDAT : 

•  Freeze  axes  in  3D  Animator  to  view  actual  and  perceived  motions  in  one  or  two  axes 
alone. 

•  Explore  a  ‘batch’  execution  mode  that  allows  the  user  to  select  a  range  of  values  for  the 
various  SD  conditions  (e.g.,  roll  threshold  of  1.1  to  1.5  in  increments  of  0.1).  The  issues 
for  such  a  version  include  control  of  combinations  and  permutations  and  how  the  output 
would  be  presented. 

•  Save  SDAT  outputs  in  Excel  spreadsheets,  with  an  analysis  as  a  single  workbook  and 
each  execution  as  a  worksheet  within  the  workbook. 

•  Automatic  interpolation  for  data  files  with  blank  cells  of  data  files.  The  manual 
interpolation  process  is  laborious. 

•  Re-develop  SDAT  to  run  in  Linux.  Many  users  prefer  not  to  use  Windows  based 
applications,  so  a  Linux  version  would  expand  our  compatibility  and  reach  more  potential 
users. 

4.6  Applications  to  Other  Topic  Areas 

The  focus  of  this  project  has  been  on  modeling  and  supporting  pilots  in  situations  of  spatial 
disorientation.  However,  there  are  a  number  of  other  areas  where  either  the  SD  models  or  the 
model-based  approach  could  be  applied.  Human  orientation  of  one  kind  or  another  is  a  problem 
for  subject  areas  such  as  space  flight,  scuba  divers  and  firefighters.  It  may  be  possible  to  use  a 
similar  model-based  approach  to  develop  systems  to  support  personnel  in  such  environments. 

The  remainder  of  this  section  includes  three  topic  areas  for  future  work. 
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Human  Performance  Modeling  Accident  Investigation  System 

Human  Factors  professionals  have  long  been  part  of  accident  investigation  teams  but  have  rarely 
had  the  use  of  automated  assessment  tools.  A  group  of  researches  from  the  NTSB,  FAA,  NASA, 
AF,  Navy  and  Army  have  been  looking  into  the  use  of  human  perfonnance  models  as  part  of 
accident  investigation.  They  are  interested  in  the  creation  of  a  tool  or  tools  very  similar  to  SDAT 
that  support  investigation  teams.  In  addition  to  the  SD  models,  human  performance  could  be 
assessed  using  models  of  fatigue,  workload,  speech  rate  and  other  stressors  to  name  a  few.  It  is 
our  sense  that  this  expansion  from  SD  analyses  into  broader  human  perfonnance  issues 
represents  a  strong  potential  for  future  R&D  funding. 

Possible  Medical  Contributions 

SDAT  has  great  potential  for  the  medical  industry,  specifically  in  the  health  of  elderly  patients 
who  are  prone  to  slips  and  falls.  Falls  are  the  leading  cause  of  injury  deaths  among  individuals 
over  age  65,  and  a  contributing  factor  in  40%  of  nursing  home  admissions  (Gillespie,  2004; 
Tinettie  et  ah,  1988).  Approximately  35%  of  people  over  age  65  fall.  This  number  increases  to 
50%  for  individuals  over  age  80  (American  Geriatrics  Society  et  ah,  2001;  Chang  et  al.,  2004). 
The  direct  costs  alone  of  falls  among  the  elderly  are  expected  to  exceed  $32  billion  by  2020 
(Enzi,  2003).  Furthennore,  fear  of  falling  has  been  shown  to  limit  the  physical  and  social 
mobility,  and  independence  of  elders  (American  Geriatrics  Society  et  al.,  2001;  Chang  et  al., 
2004;  Gillespie  2004;  Whitney  et  al.,  1999). 

There  is  additional  work  to  be  done  in  this  area  simply  because  the  amount  of  research  studying 
the  causal  relationship  between  aging  and  falls  is  so  broad,  that  it  is  difficult  to  identify  specific 
data  that  can  be  of  use.  Additionally,  older  adults  are  a  very  diverse  group,  and  their 
characteristics  vary  so  much  that  it  is  difficult  to  isolate  the  specific  factors  that  might  affect  a 
fall.  Individuals  may  suffer  from  loss  of  balance  due  to  the  deterioration  of  multiple  sensors. 
While  it’s  known  that  the  more  risk  factors  an  individual  has,  the  more  likely  they  are  to  fall, 
there  is  little  understanding  of  how  specific  factors  impact  each  other  and/or  compound  the  risk 
for  falling  (American  Geriatrics  Society  et  al.,  2000;  Tinetti  et  al.,  1988). 

Another  area  to  explore  is  the  distinction  between  age-related  sensory  degradation  and  disease  as 
they  affect  a  person’s  balance.  Lord  et  al.,  (2001)  note  that  the  level  of  degradation  caused  by 
balance  disorders  tends  to  level  off,  whereas  that  caused  by  nonnal  aging  continues  to  decline. 
Determining  how,  and  at  what  rate  of  decline,  aging  will  affect  a  person’s  balance  would  benefit 
research  into  prevention  assessments  and  tools  and  methods  for  rehabilitation  and  fall 
prevention. 

Unfortunately,  balance/stability  assessments  and  fall  prevention  are  typically  reactive  measures 
that  are  addressed  after  a  person  has  already  exhibited  some  degree  of  balance  dysfunction  or  has 
already  fallen.  Additionally,  perfonnance  measures  are  generally  poor  at  discriminating  among 
those  with  the  lowest  levels  of  functioning  (Suthers  &  Seeman,  2004).  There  is  suspicion  that 
some  existing  tests  are  not  fine  grained  enough  to  pick  up  impairment  (Lord  et  al.,  2001). 

Much  research  has  been  done  in  studying  falls  and  fall  prevention  among  the  elderly.  However, 
this  research  is  varied  and  so  vast  that  it  is  difficult  to  isolate  data  that  may  be  particularly 
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relevant  for  specific  areas  of  concern.  One  key  need  is  for  a  central  repository  for  this  research 
that  allows  users  to  query  specific  risk  factors  and  circumstances  that  can  affect  a  fall. 

Finally,  in  addition  to  a  central  database  to  update  and  store  innovative  research  into  the  subject, 
more  research  needs  to  occur  outside  of  a  lab  setting.  Most  of  what  is  known  about  the  specifics 
of  falls  comes  only  from  self  reporting.  There  are  clearly  discrepancies  in  the  data  collection.  In 
one  study  of  people  with  dizziness,  24%  of  participants  had  fallen  in  the  previous  6  months. 
“Many  of  the  patients  failed  to  identify  causes  of  falling  or  to  state  whether  the  falling  was 
related  to  dizziness”  (Lin  et  al.,  2002). 

Given  these  challenges  in  the  study  of  elderly  falls,  we  feel  that  SDAT  has  some  clear  relevance 
and  could  be  used  as  a  useful  tool  to  further  examine  the  mechanics  of  a  fall.  As  previously 
stated,  one  area  we  would  like  to  explore  in  future  Phases,  is  quantifying  the  visual  contribution 
to  spatial  orientation.  Older  adults  rely  more  heavily  on  visual  cues  to  maintain  balance  than 
younger  adults  who  rely  more  on  proprioceptive  and  vestibular  cues  (Lockhart,  Smith,  & 
Woldstad,  2005).  Having  a  better  grasp  of  the  interrelationship  between  the  visual  and  vestibular 
sensors  would  allow  us  to  better  isolate  the  most  relevant  risk  factors  for  falling.  Additionally, 
establishing  a  central  repository  of  current,  relevant  research  could  enable  us  to  determine 
SDAT’s  applicability  to  balance  disorders  as  a  whole,  not  just  among  the  elder  population. 
Finally,  SDAT’s  modeling  capabilities  would  allow  us  to  vary  and  simulate  the  many  parameters 
and  contributing  factors  of  a  fall.  This  modeling  would  enable  us  to  better  isolate  those  risk 
factors  which  most  affect  a  person’s  balance,  thus  improving  the  quality  of  proactive  preventive 
measures  and  assessments  performed  by  clinicians  in  the  earliest  stages  of  sensory  degradation. 

UAV  and  General  Applications 

Given  the  prevalence  and  technological  growth  in  Unmanned  Aerial  Vehicles  (UAVs),  our 
model-based  approach  could  be  potentially  helpful  to  combat  UAV  operator  disorientation,  with 
focus  placed  on  the  visual  sources  of  SD  in  UAV  control. 

There  is  visual  dominance,  but  other  cues  impact  orientation.  Quantifying  the  visual 
contribution,  as  well  as  other  inputs  to  spatial  orientation  (e.g.,  proprioceptive)  would  ultimately 
yield  an  SO  model  for  any  application  -  UAVs  and  others.  Then,  depending  upon  the  specific 
application,  other  model  inputs  (e.g.,  the  vestibular  contribution)  could  be  added  as  needed.  If 
we  can  develop  a  sophisticated  SO  model  -  one  that  contains  visual,  vestibular  and 
somatosensory  inputs,  the  applications  could  be  endless  -  medical  (enhancing  balance),  military 
(effects  of  non-lethal  energy),  and,  commercial  applications  (improving  virtual  games)  -  to  name 
a  few.  Clearly  there  is  ample  cause  for  follow-on  research  and  development. 
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5  Summary  &  Conclusion 

Phase  II  accomplishments  were  numerous  and  significant,  and  many  were  captured  within 
submitted  journal  articles  that  will  be  published  within  the  next  year.  Improvements  to  the 
detection  subsystem  included: 

•  Improved  models  from  Phase  I  for  Leans  and  Coriolis,  a  Graveyard  Spiral  model,  and  an 
explicit  detection  of  significant  differences  between  actual  and  perceived  pitch  angle,  roll 
velocity,  and  yaw/heading  velocity  as  components  of  “General  Illusion”  detection;  and, 

•  Experiments  and  mishap  data  analyses  that  verified  and  validated  model  improvements. 

Improvements  to  the  aiding  subsystem  were  due  to: 

•  Experiments  that  verified  the  usefulness  of  our  SD  Icon; 

•  An  experiment  that  proved  the  efficacy  of  voice  and  tactile  commands,  used  in 
conjunction  with  the  SD  Icon,  to  improve  fighter  pilot  responses  to  unusual  attitudes; 
and, 

•  Testing  of  context-sensitive  multisensory  countenneasure  timing  which  included 
suppressing  audio  cues  and  commands  in  the  presence  of  a  pilot’s  high  audio  workload. 

Both  the  detection  and  aiding  subsystem  improvements  were  embodied  within  a  single  integrated 
tool  (SDAT)  that  proved  its  usefulness  to  accident  investigators.  SDAT  is  available  to  federal 
government  customers  under  standard  SBIR  contract  terms. 

Accomplishments  above  and  beyond  the  statement  of  work  requirements  included: 

•  A  US  Patent  application; 

•  Four  professional  journal  articles,  one  in  Russian  (published  Apr-Jul  05),  one  submitted 
to  ASEM,  and  two  to  IJAP; 

•  Four  professional  conference  presentations  (HCI-Aero  04,  WinterSim  04,  AIAA-ATIO 
05,  AsMA  06); 

•  SD  mishap  data  analyses  for  the  Navy  and  NTSB,  as  well  as  ‘extra’  ones  (T-38,  F-16 
HUD  tape)  for  AFSC; 

•  Preliminary  thoughts  about  applying  our  system  and  methods  to  such  diverse  applications 
as  UAVs  and  gerontology; 

The  project  team  (see  author  list  on  the  cover  and  the  Acknowledgments)  is  very  proud  of  its 
accomplishments  throughout  the  effort’s  three  years  (Phase  I  and  II),  and  we  recognize  that  there 
is  still  much  to  do  to  help  real  pilots  who  experience  SD  -  often  with  fatal  consequences.  Many 
of  us  are  pilots  who  have  experienced  SD,  and  so  we  are  eager  to  keep  contributing  to  progress 
in  preventing  the  adverse  consequences  of  spatial  disorientation.  Fatal  SD  accidents  are 
preventable,  even  if  SD  itself  is  not  entirely  preventable. 
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Appendix  A.  Publications  Appendix 


Our  first  publication  was  our  Phase  I  final  report.  An  excerpt  of  that  was  published  in  a  Russian 
journal  as:  Small,  R.L.,  Fisher,  A.M.,  Keller,  J.W.,  &  Wickens,  C.D.  (2005).  A  pilot  spatial 
disorientation  aiding  system.  In  V.  Ponomarenko  (Ed.),  International  Academy  of  Human's 
Problems  in  Aviation  and  Cosmonautics  Bulletin,  2(17),  pp.  26-45  &  2(18),  pp.  28-38. 

Next  was  the  IJAP  paper  describing  our  SD  Icon  low-fidelity  experiment  at  USAFA.  That  paper 
was  cleared  for  public  release  by  the  USAFA  Public  Affairs  office,  and  was  accepted  by  IJAP 
with  some  revisions.  Publication  should  be  imminent.  It  appears  starting  on  the  next  page  and  is 
followed  by: 

•  A  conference  paper  for  HQ-Aero  in  Toulouse,  France; 

•  After  HQ-Aero,  we  presented  a  paper  at  AIAA-ATIO.  Both  of  the  preceding  papers 
were  based  upon  infonnation  cleared  for  public  release  from  the  Phase  I  final  report; 

•  An  ASEM  article  that  describes  our  GAT  somatogyral  experiment  (on  9/28/2006 
accepted  for  publication); 

•  Another  IJAP  article  draft  for  the  recently  completed  SOAS  test;  and, 

•  An  abstract  for  an  ISAP  2007  paper. 

Two  other  conference  presentations,  which  did  not  require  papers  -  they  were  slide  presentations 
only  -  were  WinterSim  2004  and  AsMA  2006.  They  are  not  reproduced  here,  but  have  been 
delivered  to  our  customer  and  were  cleared  for  public  release.  In  the  interest  of  completeness, 
we  should  also  mention  that  we  attended  AsMA  2004  (where  we  learned  about  Creare’s 
modeling  SBIR),  and  staffed  an  exhibit  at  AsMA  2005  (where  we  met  many  of  the  key  AFSC 
people  with  whom  we  have  worked  on  data  analyses  since  then). 
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ABSTRACT 

Twenty-two  participants  (12  with  prior  flying  experience  and  12  without),  performed  a 
series  of  trials  in  a  low  fidelity  flight  simulator  in  which  they  attempted  to  recover  from  a  series 
of  unusual  attitudes.  Two  display  variables  were  examined:  (1)  the  attitude  display  was  either  a 
traditional  head-up  display  (HUD)  with  an  inside-out  motion  and  pitch  ladder,  or  an  Arc- 
Segmented  Attitude  Reference  (ASAR)  display;  and  (2)  a  command  icon  that  pointed  to  the 
appropriate  airplane  rotation  for  recovery  direction  in  pitch  and  roll  was  either  present  or  absent. 
The  results  revealed  that  the  icon  speeded  the  initial  correction  and  reduced  the  number  of  roll 
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reversal  errors,  while  the  traditional  HUD  decreased  the  total  time  to  recovery.  Experienced 
pilots  were  more  disrupted  by  the  ASAR  display  than  were  novices. 


INTRODUCTION 

Human  factors  researchers  have  investigated  different  fonns  of  aircraft  display 
symbologies  since  Sperry  and  Doolittle  performed  the  first  instrument-only  flight  in  1929.  These 
two  pioneers  of  aviation  recognized  that  the  attitude  indicator  (AI)  was  of  paramount  importance 
in  maintaining  controlled  flight  in  low  visibility  environments,  and  continued  to  improve  upon 
their  design.  Today  the  AI  is  recognized  as  perhaps  the  only  way  to  prevent  spatial 
disorientation  (SD),  which  costs  the  Department  of  Defense  over  $300  million  per  year  in 
accident  investigations,  family  compensation,  and  aircraft  reconstruction.  “On  average,  the 
USAF  [United  States  Air  Force]  alone  loses  five  aircraft  with  aircrews  (and  sometimes  with 
passengers)  each  year  due  to  spatial  disorientation,  and  most  of  these  are  related  to  loss  of 
attitude  awareness”  (Ercoline  et  ah,  2000,  p.  489).  To  help  prevent  these  mishaps,  it  is  necessary 
to  provide  the  pilot  with  the  most  intuitive  and  effective  AI  possible  (Self  et  ah,  2002). 

Two  different  types  of  approaches  can  be  taken  to  assist  in  attitude  recovery.  One 
approach,  based  on  the  status  display  design,  presents  information  regarding  the  current  attitude 
status.  However,  as  a  status  display  the  commonly  used  inside-out  moving  horizon  attitude 
indicator  has  sometimes  been  criticized  for  inducing  control  reversals,  as  pilots  confuse  the 
moving  element  on  the  display  (the  horizon)  with  the  element  in  motion  in  the  real  world  (the 
airplane)  (Roscoe,  2002;  Previc  &  Ercoline,  2001;  Kovalenko,  1991).  When  this  design  is 
incorporated  on  a  head-up  display  or  HUD  (often  the  most  critical  source  of  attitude  information 
for  the  Air  Force  pilot),  dangers  of  motion  and  attitude  confusion  can  be  amplified  because  the 
skeletal  nature  of  the  HUD  symbology  prevents  color  coding  of  ground  and  sky  (typical  of  the 
head-down  AI).  Several  approaches  have  been  implemented  on  HUDs  to  allow  better 
discrimination  of  extreme  pitch  down  from  pitch  up  attitudes  (Newman  &  Haworth,  2004). 

These  include  for  example,  dashed  (negative  pitch)  versus  solid  (positive  pitch)  markers  for  each 
rung  of  the  pitch  ladder,  or  bent  pitch  ladder  bars,  whose  angle  indicates  the  degree  of  pitch. 

As  an  alternative  to  the  traditional  Air  Force  HUD  with  its  inside-out  pitch  ladder  (Fig. 
la),  some  designers  have  proposed  the  Arc-Segmented  Attitude  Reference  (ASAR)  display 
(North,  2003;  Jenkins  et  ah,  2002;  Geiselman,  1999).  This  display  (Fig.  lb)  has  the  advantage 
that  a  level  attitude  (zero  pitch  and  roll)  is  indicated  by  a  simple  and  easily  perceptible  semi¬ 
circle  object  (a  horizontally  oriented  half  of  a  “grapefruit,”  which  has  led  to  this  being  called  the 
“grapefruit  display”).  Deviations  from  level  pitch  are  rendered  by  the  amount  of  arc  displayed. 
Deviations  in  bank  are  rendered  by  whether  the  center  of  the  arc  is  rolled  left  or  right.  Thus 
when  the  grapefruit  disappears  entirely,  it  corresponds  to  a  direct  sky  pointing  attitude  (90°  nose 
up;  just  as  the  world  would  disappear  entirely  in  the  pilot’s  forward  view).  When  the  grapefruit 
forms  a  complete  circle,  it  corresponds  to  a  direct  ground  pointing  attitude  (90°  nose  down;  just 
as  the  full  circle  of  the  earth  would  be  rendered  looking  directly  at  it  from  space).  In  any  state  in 
between,  an  arc  is  formed  whose  direction  corresponds  to  bank  (Newman  &  Haworth,  2004). 

The  roll  information  acts  similarly  to  the  traditional  HUD  symbology:  as  the  actual  aircraft  rolls 
counterclockwise,  the  arc  appears  to  rotate  clockwise.  In  the  example  shown  in  Figure  lb,  the 
aircraft  is  pitched  up  (because  the  arc  length  is  short)  and  rolled  right. 
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While  the  ASAR  display  is  appealing  in  its  concept,  few  objective  evaluations  have  been 
accomplished  which  compare  and  contrast  it  with  more  traditional  HUDs  for  attitude  recovery. 
Those  that  have  been  performed  have  had  inconsistent  outcomes,  some  favoring  the  ASAR 
(Geiselman,  1999;  Jenkins  et  ah,  2002;  Ercoline  et  ah,  2002),  and  some  not  (Davy,  Dudfield  and 
Campbell,  1997).  Reasons  for  this  inconsistency  remain  varied.  One  problem  is  that  the  relative 
novelty  of  the  ASAR  may  conflict  with  the  greater  familiarity  of  the  traditional  HUD  for 
experienced  pilots.  Another  is  that,  when  presented  as  a  HUD,  the  ASAR  presents  attitude  in  a 
very  different  frame  of  reference  from  that  of  the  visual  world  beyond;  whereas  in  the  inside-out 
(IO)  version,  the  two  reference  frames  are  identical  and  aligned  (i.e.,  “conformal”).  This 
alignment  would  not  of  course  be  characteristic  were  the  display  represented  in  an  off-boresight 
helmet  mounted  display  (HMD),  and  explains  why  the  greater  benefits  to  the  ASAR  appear  to 
emerge  when  it  is  evaluated  as  a  candidate  for  off-boresight  viewing  in  an  HMD  (Jenkins  et  ah, 
2002;  Ercoline  et  al.,  2002). 


(b) 

Figure  1.  (a)  F-16  IO  display  and  (b)  ASAR  display.  Both  displays  show  the  airplane  rolled  right  45°  and 

pitched  up  60°. 

The  second,  complementary  (and  not  mutually  exclusive)  approach  to  unusual  attitude 
recovery  is  that  of  a  command  display  that  provides  direct  instructions  to  the  pilot  on  how  to 
point  the  aircraft  to  recover  a  level  attitude  from  an  unusual  attitude.  Such  commands  might  use 
a  synthetic  voice,  and  have  also  been  investigated  by  implementing  various  visual  pointing 
devices.  For  example  some  HUDs  have  included  an  “Auggie  arrow”  (named  after  its  inventor, 
Bill  Augustine;  that  points  in  the  roll  direction  to  the  horizon  (Newman  &  Haworth,  2004). 

Also,  an  “articulated  pitch  ladder”  will  bend  in  such  a  way  that  the  two  limbs  on  both  sides  of  the 
boresight  will  point  toward  the  horizon  (Newman  &  Hawarth,  2004),  such  that  raising  or 
lowering  pitch  in  response  to  the  visual  command  will  restore  a  level  attitude.  Rather  than 
embedding  such  command  information  in  the  pitch  ladder  itself,  Bainbridge  (1999)  has 
suggested  a  command  icon  using  a  simple  geometric  shape  that  is  easily  interpreted.  We  chose  a 
diamond  to  represent  pitch  and  roll  (Fig.  2).  This  icon  can  be  distorted  in  shape  such  that  the 
more  acute  corner  points  in  the  direction  (vertical  and  or  horizontal)  for  which  a  correction  is 
required  to  restore  a  level  attitude.  Figure  3  shows  our  SD  icon  superimposed  on  an  F-16  IO  and 
on  an  ASAR  display.  To  our  knowledge,  this  particular  format  of  command  display  has  not  been 
evaluated  as  a  means  of  attitude  recovery. 


(a) 
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Figure  2.  SD  icon  snapshots  for  an  aircraft  whose  state  is:  (a)  0°  Pitch,  0°  Roll;  (b)  -90°  (down)  Pitch,  0°Roll; 
(c)  0°  Pitch,  -45°  (left)  Roll;  and,  (d)  -45°  Pitch,  +45°Roll.  The  icon  in  the  figure  points  in  the  direction(s)  to 

correct  for  these  states. 


Figure  3.  (a)  F-16  IO  display  with  icon  and  (b)  ASAR  display  with  icon.  Both  displays  show  the  airplane 

rolled  left  75°  and  pitched  down  45°. 

Although  separate  vertical  and  horizontal  arrows  could  be  used,  instead  of  the  icon,  two 
facets  of  the  icon  suggest  that  it  should  be  a  superior  representation.  First,  as  a  single  geometric 
object,  it  is  more  likely  to  capture  attention  than  the  smaller  arrows.  Second,  the  two  dimensions 
are  integrated  in  one  object,  and  hence,  research  suggests  that  users  are  more  able  to  perceive  the 
joint  implications  of  both,  should  a  correction  in  both  axes  be  required  (Beringer  and  Chrisman, 
1991;  Carswell  and  Wickens,  1987;  Wickens  and  Carswell,  1995). 

In  the  current  experiment,  we  examine  the  efficacy  of  both  FIUD  design  (F-16  IO  and 
ASAR)  and  the  pointer  icon  in  restoring  a  level  attitude  from  an  unusual  one,  in  a  low  fidelity 
simulation,  using  both  experienced  and  novice  pilots. 


METHODS 

Participants 

The  study  utilized  two  groups  of  participants,  a  novice  and  an  experienced  group.  The 
novice  group  was  comprised  of  15  Behavioral  Science  students  from  the  US  Air  Force  Academy. 
The  ages  of  the  novice  participants  ranged  from  17-23  and  included  4  females  and  1 1  males. 
Eight  of  the  novice  participants  had  some  training  in  AIs  or  aircraft  operations.  The  experienced 
group  was  comprised  of  seven  cadets  who  were  soaring  instructor  pilots,  cadet  aviation 
instructors,  or  who  held  a  private  pilot  license  with  more  than  100  hours  of  flight  experience. 
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The  mean  age  of  participants  in  the  expert  group  was  22.5  years,  and  all  participants  in  this 
group  were  male. 

Materials 

The  test  apparatus  consisted  of  a  17-inch  computer  monitor  which  presented  the  image 
from  the  Dell  OptiPlex  GX200  with  Intel  Pentium  3  processor.  The  joystick  was  a  CH  Products 
F-16  flight  stick.  The  program  is  known  as  the  Flight  Perfonnance  Assessment  Simulation 
System  (F-PASS),  programmed  by  NTI,  Inc.  of  Dayton,  OH.  The  F-PASS  simulates  an 
American  F-16  fighter  aircraft;  flight  data  are  recorded  at  20  Hz. 

Procedure  and  Design 

As  participants  entered  the  lab,  they  were  first  presented  with  their  infonned  consent 
documents  and  the  participant’s  bill  of  rights.  Once  given  a  chance  to  examine  the  documents 
and  subsequently  sign  them,  the  participants  were  given  a  pretest  survey  designed  to  gather 
demographic  information  and  to  gauge  their  level  of  experience  with  using  attitude  indicators. 
Next,  the  purpose  of  the  study  was  briefed.  Practice  trials  were  administered  once  the 
participants  felt  comfortable  with  the  way  each  display  responded  and  reacted  to  stick  inputs. 
During  the  “free-flighf  ’  exposure  to  the  experimental  conditions,  the  practice  trials  consisted  of 
two  simulated  daytime  trials  in  which  computer-generated  terrain  was  visible  beyond  the  attitude 
display  on  the  screen,  and  eight  trials  in  which  only  the  display  symbology  was  visible  on  a  black 
background.  These  conditions  represented  visual  meteorological  conditions  in  the  day,  and 
instrument  meteorological  conditions  at  night,  respectively.  All  attitudes  were  reflected  in  the 
practice  trials.  Immediately  after  the  participants  finished  the  practice  trials,  they  were  exposed 
to  one  of  the  four  display  formats  (combination  of  icon  presence  and  IO/F-16  or  ASAR  HUD 
format),  and  received  eight  unusual  attitude  recovery  trials.  In  each  trial,  the  participant  was 
looking  at  a  blank  display,  when  the  attitude  indicator  appeared  representing  the  aircraft  in  an 
unusual  attitude.  Each  unusual  attitude  consisted  of  one  of  two  possible  values  of  pitch  (+  or  - 
45°)  and  four  possible  values  of  roll  (90°,  75°,  60°  and  45°).  The  latter  were  randomly  presented 
to  the  left  or  right,  and  the  order  of  pitch  and  roll  attitude  combinations  was  quasi-randomly 
presented.  Participants  were  instructed  to  provide  roll  input  to  level  the  wings,  and  then  correct 
their  pitch  attitude.  Following  the  presentation  of  the  eight  recovery  trials  with  a  given  display 
format,  instructions  were  presented  for  the  next  type  of  format  for  the  participant,  and  this 
process  was  repeated  until  all  four  fonnats  were  experienced.  Each  participant  received  all  four 
formats  in  a  counterbalanced  order. 

Three  dependent  measures  were  recorded  on  each  trial:  (1)  the  time  to  initial  stick  input, 
(2)  number  of  roll  reversal  errors  (RREs),  and  (3)  time  to  full  recovery  to  straight  and  level  flight 
(where  “straight  and  level”  was  defined  as  the  simultaneous  occurrence  of  less  than  5  degrees  of 
both  pitch  and  roll).  Each  participant’s  measures  were  calculated  for  each  trial,  but  then 
averaged  across  the  eight  unusual  attitudes  to  generate  three  single  measures  for  each  participant. 
Participants  were  also  asked  to  subjectively  rate  on  a  5-point  scale  each  display  format  according 
to  ease  of  use,  intuitiveness,  and  if  it  effectively  helped  them  perform  the  task.  They  were  then 
asked  to  rank  order  the  four  display  fonnats  by  preference  and  to  rate  if  the  addition  of  the  icon 
helped  them  to  effectively  accomplish  their  task. 
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RESULTS 

A  multivariate  analysis  of  variance  (MANOVA)  first  examined  the  overall  significance 
of  the  full  2  (experience)  x  2  (display)  x  2  (icon)  design  for  all  dependent  measures.  Results 
from  the  MANOVA  indicated  a  significant  difference  for  display  format,  F(3,18)  =  21.09,  p  < 
0.001  and  icon  presence,  F(3,18)  =  10.78,  p  <  0.001.  Although  the  average  data  of  the 
experienced  group  showed  better  performance  across  all  dependent  measures,  the  MANOVA  did 
not  indicate  a  significant  difference  between  the  two  groups,  F(3,18)  =  1.427,  p  >  0.10. 

However,  there  was  a  marginally  significant  effect  for  the  experience  x  display  interaction, 
F(3,18)  =  2.71,  p  =  0.075,  indicating  experts  and  novices  were  affected  differentially  by  the 
ASAR  and  IO  displays.  Because  the  MANOVA  test  did  not  indicate  any  significant  (p  <  0.05) 
differences  between  experts  and  novices  in  the  effects  of  the  two  dependent  variables,  the  data 
for  both  groups  were  combined  and  analyzed  using  univariate  ANOVA  tests  for  the  2  (display)  x 
2  (icon)  design.  The  full  data  set  for  all  participants  (expert  and  novice)  is  provided  in  Table  Al- 
1. 


Table  L  Descriptive  statistics  for  all  participants  (n=22). 


Measure 

Icon 

Input 

RRE 

Recovery 

M 

SD 

M 

SD 

M 

SD 

ASAR 

SD  Icon 

0.9941 

0.29 

0.7273 

1.39 

12.6127 

2.72 

No  SD  Icon 

1.2873 

0.78 

2.3182 

1.94 

12.9418 

3.19 

IO 

SD  Icon 

1.0323 

0.39 

0.3636 

0.79 

10.5345 

3.10 

No  SD  Icon 

1.3314 

0.71 

1.5909 

1.44 

11.1373 

3.09 

Initial  Stick  Input 

The  initial  stick  input  was  0.3  seconds  faster  with  the  icon  than  without,  F(l,21)  =  10.03, 
p  <  0.01  but  was  not  influenced  by  display  format,  nor  did  the  two  independent  variables 
interact. 

Roll  Reversal  Errors 

As  illustrated  in  Figure  4,  the  number  of  roll  reversal  errors  was  also  substantially 
reduced  by  the  icon’s  presence,  F(1 ,2 1)  =  33.58,  p  <  0.001  to  a  value  of  one-third  of  that  when 
the  icon  was  absent.  Display  format  provided  a  marginally  significant  benefit  by  reducing  the 
number  of  roll  reversal  errors,  F(  1 ,2 1)  =  4.23,  p  =  0.05.  The  mean  number  of  roll  reversal  errors 
was  slightly  lower  with  the  IO  display  than  with  the  ASAR  display.  There  was  no  interaction  of 
display  format  with  icon  presence. 
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Figure  4.  Number  of  roll  reversal  errors  as  a  function  of  display  format  and  icon  presence. 

Time  to  Full  Recovery 

Finally,  the  total  time  to  full  recovery,  while  not  influenced  by  the  presence  of  the  icon, 
was  significantly  influenced  by  display  format,  being  approximately  1 .9  sec  faster  with  the  IO 
HUD,  than  with  the  ASAR  HUD,  F(l,21)  =  35.47,  p  <  0.001.  Again,  there  was  no  interaction 
between  the  two  independent  variables. 

Experience  x  Display  Interaction 

As  indicated  in  the  MANOVA  test,  there  was  a  differential  effect  of  display  format  for 
experts  vs.  novices.  Conducting  a  univariate  ANOVA  on  the  experience  x  display  format 
interaction  showed  a  significant  result  for  the  time  to  full  recovery  measure,  F(l,20)  =  6.52, 
p  <  0.05.  Expert  recovery  time  was  more  severely  slowed  by  the  ASAR  display  (3.03  seconds 
slower  than  the  IO  display)  when  compared  to  novices  (1.45  seconds  slower  than  the  IO  display). 
This  difference  is  shown  in  Figure  5. 
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Figure  5.  Time  to  full  recovery  as  a  function  of  display  format  and  experience. 
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Subjective  Ratings 

The  subjective  ratings  for  ease,  intuitiveness,  and  effectiveness  at  completing  the  task  all 
followed  the  same  trends,  as  shown  in  Figure  6.  The  traditional  (F-16  10)  HUD  with  the  icon 
consistently  rated  the  highest,  followed  closely  by  the  ASAR  with  icon.  The  ASAR  alone  was 
by  far  the  least  preferred  display  format.  Only  three  out  of  the  22  subjects  did  not  feel  that  the 
addition  of  the  icon  helped  them  perform  their  task  effectively;  the  overall  rating  was  a  4. 16  out 
of  5. 


5=  strongly  agree 
4=  agree 
3=  neutral 
2=  disagree 
1=  strongly  disagree 


IO/ICON 


ASAR 
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Figure  6.  Subjective  ratings  of  the  four  combinations  of  display  format  and  icon. 


DISCUSSION 

The  results  of  the  current  study  are  fairly  straightforward,  with  nearly  identical  effects 
observed  for  both  novices  and  higher  time  pilots  (although  neither  group  had  flown  with  HUDs). 
For  both  groups,  the  icon  was  beneficial  in  speeding  the  initiation  of  the  original  correction,  and 
in  assuring  that  corrections  were  in  the  appropriate  direction  in  the  roll  axis.  Such  an  advantage 
of  presenting  the  unambiguous  command  icon  implicitly  signals  some  of  the  problems  with  pure 
status  attitude  information  (the  non-icon  conditions)  that  have  been  associated  with  the 
ambiguity  of  the  artificial  horizon  (Previc  &  Ercoline,  1999;  Roscoe,  2002;  Wickens,  2003), 
violating  as  it  does  the  principle  of  the  moving  part  (Roscoe,  1968).  This  would  certainly  be  the 
case  for  the  F-16  HUD  display,  in  which  an  “inside  out”  motion  of  the  display  symbology  is 
used.  This  motion  stereotype  is  also  violated  with  the  ASAR  display,  since  the  moving  element 
on  the  display  does  not  correspond  to  the  airplane.  Other  features  of  this  display,  discussed 
below,  also  may  have  hampered  attitude  recovery,  and  hence  led  to  the  observed  benefit  of  the 
icon. 


Both  the  performance  and  the  subjective  data  clearly  indicate  the  advantage  of  the  icon 
display  as  a  command  display.  By  now,  a  growing  body  of  research  has  documented  the 
advantages  of  command  displays  (over  status  displays)  as  decision  aids  in  high  time-stress 
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environments  (Crocoll  &  Coury,  1990;  Sarter  &  Schroeder,  2001;  Wickens,  2003).  These 
displays  eliminate  the  cognitive  step  of  inferring  the  necessary  action  from  the  observed  status, 
by  directly  presenting  that  action.  Clearly,  recovery  from  an  unusual  attitude  is  a  prototype  of 
such  a  time-stress  situation. 

Somewhat  more  puzzling  was  the  clear  emergence  of  a  cost  for  the  ASAR  display, 
relative  to  the  F-16  (IO)  HUD  symbology,  as  measured  in  total  recovery  time  and  in  roll  reversal 
errors.  While  neither  pilot  group  was  familiar  with  HUD  symbology  per  se,  we  can  possibly 
relate  the  ASAR  cost  to  the  phenomenon  of  negative  transfer  of  motion  stereotypes.  Members  of 
both  groups,  and  particularly  the  expert  group,  had  some  experience  flying  with  the  standard  IO 
moving  horizon  attitude  display.  The  F-16  HUD  symbology  we  used  preserves  this  motion, 
whereas  the  ASAR  symbology  is  less  consistent,  presenting  a  motion  pattern  that  does  not 
conform  to  the  view  forward  from  within  the  cockpit.  Furthennore,  we  might  expect  that 
symbology  to  hinder  more  those  with  greater  experience  on  the  inside-out  motion  pattern,  and 
indeed  this  expectation  was  confirmed,  as  the  experts  showed  a  full  three-second  recovery  time 
cost,  whereas  the  novice  cost  was  less  than  half  of  that. 

How  then  does  this  clear  finding  of  an  ASAR  cost  converge  with  the  various  examples  of 
ASAR  benefits  discussed  above  (Geisehnan,  1999;  Jenkins  et  al.,  2002;  Ercoline  et  al.,  2002)? 
The  answer,  presumably,  is  that  those  benefits  had  been  observed  primarily  when  the  ASAR  was 
used  as  a  component  of  a  head-mounted  display  with  off-boresight  viewing.  In  these  instances, 
the  relation  between  display  movement  and  airplane  movement  is  decoupled  in  any  case,  so 
motion  confonnity  of  rotation  between  the  airplane  and  the  displayed  airplane  symbols  become 
less  critical;  and  indeed  it  may  be  better  to  have  the  more  cognitively  compatible  moving 
airplane  display  (Wickens  et  al.,  2005).  It  is  when  the  display  and  airplane  axes  are  aligned,  as  in 
the  current  study,  that  the  benefits  of  the  same  reference  frame  (moving  airplane)  are  more  likely 
to  emerge. 


IMPLICATIONS  AND  APPLICATIONS 

The  two  implications  of  this  experiment  are  fairly  straightforward.  First,  a  command 
icon  can  be  an  extremely  useful  tool  for  attitude  recovery,  so  long  as  it  is  presented  in  a  manner 
that  is  clearly  distinguishable  from  any  status  information  (Wickens,  2003).  Second,  the  ASAR 
display,  with  its  relatively  unconventional  attitude  representation  may  be  effective  for  off-axis 
viewing  in  a  helmet-mounted  display,  but  probably  is  not  advisable  for  standard  forward-looking 
HUD  symbology.  This  issue  of  how  to  resolve  these  two  possibly  conflicting  guidelines  is 
important,  but  beyond  the  scope  of  the  current  paper,  and  is  addressed  in  detail  by  Wickens  et  al. 
(2004). 


Naturally  the  current  results  need  to  be  accepted  with  some  caution  as  to  their 
generalizability,  given  the  lack  of  flight  experience  in  combat  aircraft  of  all  participants,  and 
given  the  low  fidelity  of  the  simulation  used.  Another  important  issue  concerning 
generalizability  relates  to  the  paradigm  used  here,  with  an  “instantaneous”  loss  of  spatial 
orientation  rather  than,  as  in  normal  flight  conditions,  circumstances  in  which  it  may  take  several 
seconds  to  evolve  from  an  “oriented”  to  a  “disoriented”  attitude.  We  justify  our  paradigm  choice 
here  on  two  grounds.  First,  much  spatial  disorientation  is  categorized  as  “type  1”  when  the 
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orientation  loss  is  unconscious  (that  is,  the  pilot  is  unaware  of  the  loss);  therefore  when  the  pilot 
abruptly  does  transition  to  “type  2”  (becomes  aware),  this  transition  is  typically  an  abrupt  one, 
triggered  by  a  sudden  glance  at  the  instruments.  Second,  it  is  also  the  case  that  the  loss  of  spatial 
orientation  happens  when  the  pilot  is  engaged  in  head-down  concurrent  tasks,  and  so  again, 
realization  that  orientation  is  loss,  occurs  with  a  discrete  glance  at  the  instruments  -  essentially 
replicating  the  procedure  of  the  current  experiment. 

As  part  of  an  Air  Force  program  to  reduce  SD-caused  mishaps,  the  command  icon 
represents  a  significantly  helpful  step  and  should  be  further  tested  as  a  next  step  in  actual  flight 
with  Air  Force  rated  pilots.  Furthermore,  it  is  important  to  note  that  the  SD  icon  is  just  one  small 
part  of  an  overall  effort  to  reduce  the  adverse  consequences  of  SD.  The  overall  effort  should 
include:  pilot  selection;  pilot  training;  pre-mission  briefings  to  review  the  risks  of  SD;  flight  data 
collection;  and,  post-flight  analyses  to  determine  the  actual  conditions  under  which  pilots  are 
more  susceptible  to  SD.  Post- flight  analyses  will  also  suggest  which  SD  countermeasures  are 
most  effective  during  actual  missions  as  research  results  transition  into  operational  use.  Further 
details  regarding  a  more  comprehensive  approach  to  reducing  SD-caused  mishaps  are  in  Small  et 
al.  (2005). 


ACKNOWLEDGMENTS 

The  views  expressed  are  those  of  the  authors  and  do  not  reflect  the  official  policy  or 
position  of  the  US  Air  Force,  Department  of  Defense,  or  the  US  Government.  Work  was 
supported  in  part  by  US  Air  Force  contracts  F33615-03-M-6360  and  FA8650-04-C-6457;  Dr. 
Kristen  Liggett  was  our  customer.  The  authors  thank  Cadet  David  Repp  for  his  initial  work  in 
supporting  this  research. 


REFERENCES 

Bainbridge,  L.  (1999).  Processes  underlying  human  performance.  In  D.J.  Garland,  J.A.  Wise,  & 

V.D.  Hopkin  (Eds.),  Handbook  of  Aviation  Human  Factors.  Mahwah,  NJ:  Lawrence  Erlbaum 

Associates,  107-171. 

Beringer,  D.B.  and  Chrisman,  S.E.  (1991).  Peripheral  polar  graphics  display  for  signal/failure 

detection.  The  International  Journal  of  Aviation  Psychology,  7(2),  133-148. 

Carswell,  C.M.,  &  Wickens,  C.D.  (1987).  Information  integration  and  the  object  display:  An 
interaction  of  task  demands  and  display  superiority.  Ergonomics,  30(3),  511-527. 

Crocoll,  W.M.,  &  Coury,  B.G.  (1990).  Status  or  recommendation:  Selecting  the  type  of 

information  for  decision  aiding.  In  Proceedings  34th  Annual  Meeting  of  the  Human  Factors 
Society.  Santa  Monica,  CA:  HFS,  1524-1538. 

Davy,  E.C.,  Dudfield,  H.J.,  &  Campbell,  J.T.  (1997).  A  review  of  attitude  symbology  research 
for  helmet-mounted  displays.  In  R.  Jensen  (Ed.),  Proceedings  of  the  International 
Symposium  on  Aviation  Psychology.  Columbus,  OH:  The  Ohio  State  University,  287-293. 

Ercoline,  W.R.,  DeVilbiss,  C.A.,  Yauch,  D.W.,  &  Brown,  D.L.  (2000).  Post-roll  effects  on 
attitude  perception:  The  Gillingham  illusion.  Aviation,  Space,  and  Environmental  Medicine, 
71(5),  489-495. 


71 


Ercoline,  W.R.,  Self,  B.P.,  &  Matthews,  R.  (2002).  Effects  of  three  helmet-mounted  display 
symbologies  on  unusual  attitude  recognition  and  recovery.  Aviation,  Space,  and 
Environmental  Medicine,  73(11),  1053-1058. 

Geiselman,  E.E.  (1999).  Development  of  a  non-distributed  flight  reference  symbology  for 
helmet-mounted  display  use  during  off-boresight  viewing.  In  Proceedings  of  the  4th  Annual 
Situational  awareness  in  the  tactical  air  environment,  Piney  Point,  MD,  118-127. 

Jenkins,  J.C.,  Thurling,  A.J.,  Havig,  P.R.,  &  Geiselman,  E.E.  (2002).  Flight  test  evaluation  of 
the  nondistributed  flight  reference  off-boresight  helmet-mounted  display  symbology.  In  C.E. 
Rash,  &  C.E.  Reese  (Eds.),  Proceedings  of  the  SPIE  Helmet-  and  Head-Mounted  Displays 
VII,  341-355. 

Newman,  R.L.,  &  Haworth,  L.A.  (2004).  Flight  displays  II:  Head-up  and  helmet-mounted 
displays.  In  F.H.  Previc  &  W.R.  Ercoline  (Eds.),  Spatial  disorientation  in  aviation.  Reston, 
VA:  AIAA,  451-507. 

Kovalenko,  P.A.  (1991).  Psychological  aspects  of  pilot  spatial  orientation.  ICAO  Journal, 

46(3),  18-23. 

Previc,  F.H.,  &  Ercoline,  W.R.  (1999).  The  “outside-in”  attitude  indicator  revisited. 
International  Journal  of  Aviation  Psychology,  9(4),  377-401. 

Previc,  F.H.,  &  Ercoline,  W.R.  (2001).  Trends  in  spatial  disorientation  research.  Aviation, 
Space,  and  Environmental  Medicine,  72(11),  1048-1050. 

Roscoe,  S.N.  (1968).  Airborne  displays  for  flight  and  navigation.  Human  Factors,  10,  321-332. 

Roscoe,  S.N.  (2002).  Ergavionics:  Designing  the  job  of  flying  an  airplane.  International 
Journal  of  Aviation  Psychology,  12(4),  331-339. 

Sarter,  N.  &  Schroeder,  B.  (2001).  Supporting  decision  making  and  action  selection  under  time 
pressure  and  uncertainty:  The  case  of  in-flight  icing.  Human  Factors,  43(4),  573-583. 

Self,  B.P.,  Breun,  M.,  Feldt,  B.,  Perry,  C.,  &  Ercoline,  W.R.  (2002).  Longitudinal  assessment  of 
pilot  performance  using  a  moving  horizon  (inside-out),  a  moving  aircraft  (outside-in),  and  an 
arc-segmented  attitude  reference  display.  In  Proceedings  of  the  NATO  Symposium,  Spatial 
Disorientation  in  Military  Vehicles.  A  Coruna,  Spain:  NATO. 

Small,  R.L.,  Fisher,  A.M.,  Keller,  J.W.,  &  Wickens,  C.D.  (2005).  A  pilot  spatial  orientation  aiding 
system  (AIAA  paper  2005-7431).  In  Proceedings  of  the  AIAA  5th  ATIO  and  16th  Lighter-Than-Air 
Sys  Tech,  and  Balloon  Systems  Conference.  Arlington,  VA:  AIAA,  1-15. 

Wickens,  C.D.  (2003).  Aviation  displays.  In  P.  Tsang  &  M.  Vidulich  (Eds.),  Principles  and 
Practices  of  Aviation  Psychology  (Chapter  5,  147-200).  MahwahNJ:  Erlbaum. 

Wickens,  C.D.,  &  Carswell,  C.M.  (1995).  The  proximity  compatibility  principle:  Its 

psychological  foundation  and  relevance  to  display  design.  Human  Factors,  37(3),  473-494. 

Wickens,  C.D.,  Vincow,  M.,  &  Yeh,  M.  (2005).  Design  applications  of  visual  spatial  thinking. 
In  A.  Miyaki ,  &  P.  Shah  (Eds.),  Handbook  of  Visual  Spatial  Thinking.  Oxford  University 
Press. 

Wickens,  C.D.,  Ververs,  P.,  &  Fadden,  S.  (2004).  Head-up  display  design.  In  D.  Harris  (Ed.), 
Human  factors  for  civil  flight  deck  design.  Burlington,  VT:  Ashgate,  103-140. 


72 


Multisensory  Integration 

Ron  Small,  Alia  Oster,  John  Keller 

Micro  Analysis  &  Design 
4949  Pearl  East  Circle,  Suite  200 
Boulder,  CO  80301,  USA 
rsmall@maad.com 

Spatial  disorientation  (SD)  is  a  normal  human  response 
to  accelerations  in  flight.  Its  cost  to  the  US  military  is 
over  $300  million  per  year,  with  comparable  costs  to 
US  civil  aviation.  Despite  significantly  increased 
research  over  the  past  decade,  the  rate  of  accidents 
caused  by  SD  has  not  decreased.  While  the  most 
recent  research  emphases  have  been  on  understanding 
the  physiology  of  SD,  the  translation  of  the  new 
knowledge  into  tools  (e.g.,  training,  displays, 
automation)  that  help  pilots  avoid  SD  and  minimize  its 
effects  if  it  does  occur,  has  not  occurred.  Our  research 
goals  were  to  apply  multisensory  countermeasures 
(CMs)  to  SD  based  on  human  sensory  models  and  the 
pilot’s  workload.  We  assert  that  multisensory  CMs, 
applied  in  an  intelligent  fashion,  are  needed  and 
possible. 

Spatial  disorientation,  multisensory  displays,  intelligent 
aiding 

INTRODUCTION 

Spatial  disorientation,  simply  referred  to  as  vertigo 
until  becoming  the  subject  of  scientific  research,  is  a 
normal  response  to  physiologically  confusing  stimuli 
in  flight.  Humans  evolved  to  perceive  motions  on  land 
relative  to  the  earth’s  surface.  In  flight,  human  senses 
can  give  the  brain  erroneous  orientation  information. 
Everyone  who  flies  will  eventually  experience  some 
SD  illusions;  some  pilots  will  experience  them  more 
acutely  than  others. 

The  seriousness  of  SD  phenomena  to  the  flight 
environment  cannot  be  overstated.  Of  323  of  the  most 
serious  mishaps  in  the  US  Air  Force  from  1991-2000, 
20.2%  were  SD  related  [1],  Further,  it  is  estimated  that 
89%  of  general  aviation  SD  events  are  fatal  [2], 

Holmes  et  al.  [3]  have  estimated  that  roughly  80%  of 
pilots  have  experienced  SD  incidents,  although  these 
usually  do  not  lead  to  mishaps.  SD  mishaps  are  often 
fatal,  accounting  for  the  loss  of  about  40  lives  per  year 
in  the  US  military  [4,  5],  with  a  cost  of  over  $300 
million  per  year;  US  civil  aviation  has  comparable  SD 
costs.  Increasing  research  has  not  decreased  the  rate  of 
accidents  caused  by  SD  [6]. 

The  conflict  between  one’s  senses  and  reality  typically 
lead  to  SD  illusions,  and  has  received  much  recent 
research  attention.  Understanding  the  physiological 
causes  of  SD  is  important  to  solving  the  problem.  For 
example,  how  much  do  optical  illusions  contribute  to 


for  Pilot  Spatial  Orientation 

Christopher  D.  Wickens,  Ph.D. 

University  of  Illinois 
1  Airport  Road 
Savoy,  IL  61874  USA 


SD-related  accidents?  How  effective  is  a  particular 
attitude  display  in  helping  to  maintain  spatial  ori¬ 
entation  (SO)?  Some  answers  are  beginning  to 
emerge.  For  example,  Gomez  [7]  asserts,  “90%  of  all 
inputs  to  the  brain  that  give  the  sense  of  orientation  are 
visually  acquired.”  This  assertion  implies  that  SO 
displays  should  be  visual,  as  opposed  to  auditory  or 
tactile  displays,  which,  presumably,  contribute  only 
10%  of  the  orientation  inputs  to  the  brain.  However, 
the  SD  problem  is  too  important  to  rely  on  only  one 
sensory  channel.  Also,  because  SD  is  often  due  to  a 
sensory  conflict  or  distraction,  using  only  one  sense 
(vision)  to  combat  it  is  ill  advised.  A  multisensory 
approach  is  needed. 

For  our  purposes,  we  define  SD  as  a  loss  of  attitude 
awareness,  which  is  a  subset  of  one  widely  accepted 
definition  of  SD  [see  8].  Often  this  loss  of  attitude 
awareness  is  caused  by  degraded  visual  information 
regarding  the  true  horizon,  plus  the  input  of  misleading 
vestibular  (inner  ear)  signals.  Therefore,  we  focus  on 
models  of  vestibular-based  illusions,  the  Leans  and 
Coriolis  illusions.  For  brevity,  we  only  describe  our 
Leans  model.  For  details  about  Coriolis  and  other 
illusions,  as  well  as  general  information  about  SD  and 
SO,  see  [9]. 

The  most  common  SD  illusion  is  the  Leans  [3,  8], 
which  usually  entails  an  erroneous  feeling  of  roll.  A 
typical  case  is  as  follows  [10]:  In  the  clouds,  the  pilot 
unknowingly  enters  a  turn  at  a  vestibular  sub-threshold 
rate  of  less  than  about  2°/second  so  that  the  semi¬ 
circular  canals  do  not  sense  the  turn.  If  the  pilot  then 
becomes  aware  of  the  aircraft’s  roll  angle  (e.g.,  by 
looking  at  the  aircraft’s  attitude  reference  information) 
and  abruptly  returns  to  level  flight,  he  or  she  will  now 
receive  a  vestibular  sensation  of  an  opposite  roll.  If  the 
pilot  continues  to  rely  upon  flight  instillments  to 
maintain  a  level  attitude,  he  or  she  may  also  lean 
toward  the  erroneously  perceived  upright  (hence  the 
illusion’s  name).  If  the  pilot  does  not  rely  upon  the 
instillments  at  this  point,  but  rather  the  (incorrect) 
vestibular  signal  of  upright,  he  or  she  will  return  the 
aircraft  to  its  original  roll  angle.  Without  awareness 
and  conscious  correction,  the  roll  may  lead  to  a  pitch 
down  attitude  and  a  loss  of  altitude,  which  may 
progress  to  the  point  of  ground  impact.  This 
progression  is  illustrated  in  Figure  1,  which  is  the 
model  of  the  Leans  that  we  use. 
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COUNTERMEASURES 

How  does/should  the  aviation  community  deal  with 
this  problem?  There  is  a  range  of  measures  to  prevent 
and  counter  SD.  Techniques  include:  pilot  selection 
and  training;  pre-mission  risk  analyses  and  awareness; 
real-time  cockpit  systems,  such  as  visual  displays, 
audio  attitude  cues,  automated  audio  recovery 
commands,  haptic  or  tactile  displays,  and  ground 
collision  avoidance  systems;  and,  post-accident 
analyses.  Much  research  has  been  done  to  enhance 
visual  displays  to  help  prevent  or  counter  SD.  Again, 
reference  [9]  thoroughly  describes  such  issues.  Our 
contribution  to  display  enhancement  builds  upon  work 
by  Bainbridge  [11],  who  recommends  simple  geo¬ 
metric  shapes  to  ease  interpretation.  We  offer  the  SD 
icon  shown  in  Figure  2,  which  would  appear  on  one  or 
more  visual  displays  under  the  control  of  an  intelligent 
cockpit  aiding  system  [12]. 


_w _ m _ (£) _ m 


Pilot 

States 


Aircraft 

States 


World 

States 


Countermeasure 

Assessor 


Countermeasures 


Cockpit  Displays 


Figure  3.  Spatial  disorientation  aiding  system 
architecture. 


Figure  4.  SD  remediation  network. 


CONCLUSION 

Our  real-time  prototype  illustrates  how  our  SD  Aiding 
System  responds  to  three  scenarios:  a  simulated  Leans 
illusion,  a  simulated  Coriolis  illusion,  and  actual  flight 
data  from  an  F-16  mishap  attributed  to  SD  by  the  Air 
Force  Safety  Center.  Our  initial  research  proved  the 
feasibility  of  using  an  intelligent  SD  aiding  system  to 
trigger  multisensory  CMs  to  help  combat  SD. 

Ongoing  research  will  validate  our  models  and  test  the 
efficacy  of  our  approach  via  pilot-in-the-loop 
experiments. 


Figure  2.  SD  icon:  (a)  0°  pitch  &  roll;  (b)  +90°  pitch,  0° 
roll;  (c)  0°  pitch,  45°  right  roll;  and,  (d)  +45°  pitch,  45° 
left  roll. 

A  MULTISENSORY  SD  AIDING  SYSTEM 

Our  approach  is  to  intelligently  apply  multisensory 
CMs  to  aid  the  pilot  in  avoiding  SD’s  bad 
consequences.  To  do  this,  we  model  the  pilot’s  SD 
illusion  severity,  the  pilot’s  workload,  and  the  severity 
of  the  resulting  flight  situation.  As  an  SD  event 
worsens  and  as  the  aircraft  enters  unusual  attitudes  that 
could  result  in  an  accident,  our  multisensory  CMs 
become  more  intrusive.  For  example,  if  we  assess  that 
the  pilot  has  a  mild  case  of  the  Leans,  but  is  close  to 
the  ground  and  visually  task-loaded,  our  intelligent  SD 
Aiding  System  triggers  intrusive  audio  and  tactile 
CMs.  As  the  pilot  recovers,  CMs  become  less 
intrusive.  If  the  situation  worsens  (e.g.,  imminent 
ground  impact),  CMs  become  more  urgent,  which 
might  include  auto-recovery.  Figure  3  shows  the 
architecture  of  our  system;  Figure  4  depicts  the 
remediation  network  used  to  determine  the 
intrusiveness  of  CMs. 
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Spatial  disorientation  (SD)  is  a  nonnal  human  response  to  accelerations  in  flight.  Its  cost  to  the 
US  military  is  over  $300  million  per  year,  with  comparable  costs  to  US  civil  aviation.  Due  to 
significantly  increased  research  over  the  past  decade,  some  progress  is  being  made  in  helping 
pilots  avoid  SD’s  adverse  consequences  (i.e.,  mishaps,  incidents,  or  accidents).  We  used  a  multi- 
sensory  model-based  approach  to  intelligently  trigger  SD  countenneasures.  Results  for  two 
simulated  and  one  actual  SD  event  were  very  promising,  and  indicate  that  the  next  research  step 
should  include  verifying  and  validating  our  models,  and  evaluating  our  spatial  orientation  aiding 
system  (SOAS)  via  pilot-in-the-loop  simulations  and  flight  tests.  SOAS  evaluates  the  aircraft 
motions,  pilot  workload,  aircraft  state,  and  world  state  to  detennine  if  the  pilot  is  probably 
suffering  from  SD,  and,  if  so,  SOAS  applies  a  range  of  countermeasures  to  assist  the  pilot  in 
recognizing  the  SD  and  recovering.  Countermeasures  include  visual  cues,  audio  cues,  audio 
recovery  commands,  tactile  cues,  and  olfactory  cues,  as  well  as  the  more  extreme  measures  of 
auto-recovery  and  auto-ejection  (to  save  the  pilots  life,  only  if  all  other  countenneasures  have 
failed). 


Introduction 

Spatial  disorientation  (SD)  is  a  nonnal  human  response  to  accelerations  in  flight.  Its  cost  to 
the  United  States  (US)  military  is  over  $300  million  per  year1’2  with  comparable  costs  to  US  civil 
aviation.3  Despite  significantly  increased  research  over  the  past  decade,4’5  the  SD-related 
accident  rate  has  not  decreased,  although  Braithwaite6  reports  some  promising  trends  for  Army 
helicopter  operations  based  upon  realistic  SD  training  sorties.  Everyone  who  flies  will  eventually 
experience  some  SD  phenomena;  some  pilots  will  experience  them  more  acutely  than  others. 

The  conflict  between  one’s  senses  and  reality,  which  is  at  the  core  of  the  SD  problem,  has 
received  much  recent  research  attention.  ’  Understanding  the  physiological  causes  of  SD  is 
essential  to  solving  the  problem,  but  is  outside  the  scope  of  this  paper;  excellent  references 

r  o  q 

exist.  ’  ’  The  inability  of  pilots  to  accurately  and  intuitively  perceive  aircraft  position  without 
reliance  upon  visual  cues  (from  flight  instruments  or  the  outside  world)  is  a  major  crux  of  the 
aviation  mishap  problem.  Maintaining  spatial  orientation  cannot  be  done  in  present-day  flight 
operations  unless  one  is  attending  to  the  appropriate  visual  cues.  Unfortunately,  many  of  an 
aviator’s  distracting  secondary  flight  tasks  are  also  of  a  visual  nature,  so  continuous  attention  to 
one’s  spatial  orientation  cannot  be  maintained  using  current  visual  displays.  Furthermore, 
attention  is  also  diverted  to  non-visual  sources;  for  example,  the  distraction  of  voice 
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communications  or  problem  solving.10  Holmes  and  colleagues11  identify  “distraction/task 
saturation”  as  an  important  factor  in  their  list  of  causes  of  spatial  disorientation.  The  problem 
concerning  the  allocation  of  limited  attentional  resources  is  compounded  by  the  fact  that 
attentional  resources  will  be  drawn  to  more  natural  and  salient  body  (vestibular)  cues  concerning 
orientation,  which  in  the  environment  of  flight  are  not  veridical.  In  other  words,  the  problem  of 
SD  in  flight  is  not  caused  merely  by  attentional  limitations;  rather,  the  problem  is  the  formation 
of  an  incorrect,  yet  persuasive,  subconscious  tendency  to  rely  upon  vestibular  orientation  cues. 

A  tendency  to  not  recognize  hazards  during  moments  of  distraction  has  been  noted  by  SD 
researchers.  The  typical  SD  mishap  occurs  when  visual  attention  is  distracted  from  the  aircraft’s 
orientation  instruments  and  the  horizon  is  not  visible  or  not  being  monitored.  “  For  example,  all 
US  Air  Force  SD  mishaps  during  1990-91  were  categorized  as  unrecognized  SD  (Type  1).  In 
other  years,  relatively  few  SD  mishaps  were  attributed  to  SD  that  was  recognized  (Type  2)  or  SD 
that  was  completely  incapacitating  (Type  3).  In  the  past,  the  aerospace  community  came  to 
realize  that  “pilot  error”  was  not  a  sufficiently  specific  description  for  the  cause  of  an  airplane 
crash.  Now  we  must  avoid  the  temptation  to  consider  our  job  complete  when  we  have  identified 
some  general  human  psychological  state  as  the  cause  of  an  accident.  As  Lyons  and  colleagues14 
have  said  concerning  SD  mishaps:  “. .  .if  both  an  attention  deficit  and  SD  are  part  of  the  chain  of 
events  leading  to  an  accident,  each  should  be  separately  identified  as  a  causal  factor  if 
elimination  of  either  would  have  prevented  the  accident.”  This  advice  makes  very  good  sense  in 
the  short  tenn.  In  the  long  term,  what  is  needed  is  to  incorporate  separately  tabulated  factors  into 
a  model  that  can  accurately  predict  the  amount  of  variance  accounted  for  by  each  factor 
contributing  to  human  error  in  flight.  This  requires  a  commitment  to  differentiate  factors  that  are 
usually  aggregated  a  priori. 

Clearly,  SD  is  a  huge  problem  for  US  military  and  civilian  aviation,  with  likely  parallels 
around  the  world.  Consequently,  it  is  a  problem  worthy  of  a  depth  and  breadth  of  research. 
Solutions  do  not  have  to  solve  the  entire  problem;  even  a  partial  solution  can  contribute  to  saving 
lives  and  aircraft.  Because  SD  is  often  due  to  a  sensory  conflict  or  distraction,  using  only  visual 
displays  to  combat  it  is  ill  advised.  A  multisensory  approach  is  needed.  Another  key  to 
combating  SD  is  to  prevent  its  adverse  consequences.  As  an  SD  event  deteriorates,  the  sequence 
of  multisensory  countermeasures  should  increase  in  intrusiveness.  To  properly  trigger  such 
countermeasure  sequences,  we  use  SD  illusion  models,  pilot  workload  models,  and  situational 
consequence  models  in  our  Spatial  Orientation  Aiding  System  (SOAS).  The  following  sections 
briefly  explain  the  illusions  we  selected  to  model,  current  and  suggested  countenneasures,  our 
multisensory  SOAS,  and  our  preliminary  results. 

Illusions 

Spatial  disorientation  accidents  have  been  referred  to  as  those  in  which  the  pilot  crashes 
because  the  seriousness  of  the  situation  does  not  become  evident  until  it  is  too  late.  There  are 
numerous,  well-documented  illusions  that  are  generated  by  visual-vestibular  conflicts.  This 
section  describes  two  illusions  that  we  studied  in  depth  and  then  modeled. 

The  most  common  SD  illusion  in  flight  is  the  Leans,11’15  which  entails  an  erroneous  feeling  of 
roll.  A  typical  case  occurs  in  the  following  scenario:16  In  poor  visibility  conditions,  the  pilot  has 
very  slowly  entered  a  turn,  probably  unknowingly,  at  a  sub-threshold  rate  of  less  than  2°/sec  (i.e., 
below  Mulder’s  constant)  so  that  the  pilot’s  vestibular  system  provides  no  sense  of  rotation.  If  the 
pilot  then  becomes  consciously  aware  of  the  aircraft’s  bank  angle  (e.g.,  by  looking  at  the 
instruments),  and  intentionally  returns  the  bank  to  the  true  level  attitude,  he  or  she  will  now 
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receive  a  vestibular  sensation  of  an  opposite  bank.  If  the  pilot  continues  to  rely  upon  flight 
instruments  to  maintain  a  level  attitude,  he  or  she  may  also  lean  in  the  orientation  of  the 
incorrectly  perceived  upright  (hence  the  illusion’s  name).  If  the  pilot  does  not  rely  upon  the 
instruments  at  this  point,  but  rather  the  intuitive  (vestibular)  signal  of  upright,  he  or  she  will 
return  the  aircraft  to  its  original  bank  angle.  Without  awareness  and  conscious  correction,  the 
bank  will  lead  to  a  gradual  pitch  down  attitude,  a  loss  of  altitude,  and  an  increase  in  airspeed.  A 
potential  side-to-side  see-sawing  process  of  correction  and  re-entry  may  continue  until  the  pilot 
is  so  disoriented  that  recovery  to  straight-and-level  flight  is  difficult,  if  not  impossible.  In  this 
case,  the  Leans  may  progress  into  a  more  severe  loss  of  altitude,  which  ends  with  ground  impact. 

The  Airman ’s  Information  Manual  describes  the  Coriolis  illusion  as  the  “most  overwhelming 
of  all  illusions  in  flight.”  It  occurs  during  “a  prolonged  constant-rate  turn  that  has  ceased 
stimulating  the. . .  [vestibular]  system,”  (i.e.,  washout)  when  there  is  an  abrupt  head  movement 
that  creates  “the  illusion  of  rotation  or  movement  in  an  entirely  different  axis”  (page  645). 17  The 
resulting  tumbling  sensation  can  be  incapacitating.  The  Coriolis  illusion  often  results  in  the 
pilot’s  loss  of  aircraft  control,  which  can  have  catastrophic  results  if  there  is  no  other  pilot  to 
assume  control.  The  power  of  the  Coriolis  illusion  to  deceive  is  further  supported  by  the  finding 
in  a  recent  survey  that  it  was  the  most  prevalent  of  those  found  in  141  pilots  attending  a  course  at 
Randolph  Air  Force  Base,  Texas  in  1997-98.  We  selected  the  Coriolis  illusion  as  the  other 
prototypical  illusion  for  our  research  because  it  is  among  the  most  common  flight  illusions  (#5  in 
the  list  compiled  in  Ref.  11)  and  is  often  fatal.  It  causes  overwhelming  disorientation  and  so  is  an 
appropriate  illusion  to  combat. 

In  summary,  the  physiological  circumstances  underlying  many  spatial  disorientation  illusions 
are  as  follows:  Aircraft  motion,  sometimes  coupled  with  head  motion,  provides  illusory 
vestibular  signals  to  the  brain.  Normally,  when  salient  visual  cues  are  available,  visual 
dominance  will  allow  the  vestibular  cues  to  signal  the  correct  orientation  and  motion.  However, 
when  the  visual  cues  are  degraded  or  missing,  the  compelling  vestibular  signals  may  dominate, 
and  illusions  may  occur.  Knowledge  of  how  these  illusions  begin  and  progress  provides 
suggestions  for  countermeasures,  which  we  address  in  the  following  section.  We  chose  the  Leans 
and  Coriolis  illusions  because  of  their  prevalence  and  because  they  represent  a  slow-onset  and  a 
quick-onset  SD  event,  respectively.  Our  assumption  was  that  if  we  could  address  both  illusions, 
we  would  expect  that  we  could  help  remedy  other  visual-vestibular  illusions. 

Countermeasures:  A  Layered  Approach 

This  section  describes  a  multisensory  approach  to  preventing,  minimizing,  or  compensating 
for  pilot  spatial  disorientation.  Since  human  spatial  orientation  has  such  a  large  visual 
component,  we  describe  visual  displays  in  some  depth.  Audio  and  tactile  displays  follow,  and 
then  an  intelligent  system  approach  to  applying  the  various  SD  countenneasures.  All  of  these 
technologies  fit  within  a  layered  approach  to  combating  SD. 

We  suggest  a  layered  approach  to  improving  pilot  attitude  awareness  and  reducing  the 
negative  consequences  of  SD  events.  Improving  current  cockpit  displays  (i.e.,  head-down,  head- 
up,  and  helmet-mounted)  and  incorporating  new  technologies  (e.g.,  audio,  tactile)  requires  a 
methodology  for  integrating  and  testing  all  of  the  salient  technologies  and  techniques  to 
determine  which  ones  are  the  best  at  enhancing  attitude  awareness,  preventing  SD  events,  and 
minimizing  the  impact  of  any  SD  events  that  do  occur.  A  layered  approach  begins  with  pilot 
training  and  mission  pre-briefings,  and  extends  through  improved  display  technologies  and 
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recovery  aiding,  into  post-flight  data  analyses  to  better  understand,  categorize,  and  track  SD 
events  to  detennine  the  actual  operational  effectiveness  of  the  various  techniques  advocated  to 
improve  SD  accident  statistics. 

Recent  research  has  explored  new  approaches  to  addressing  SD:  better  attitude  information 
for  head-down,  head-up,  and  helmet-mounted  displays;  audio  displays;  and,  tactile  displays. 
Which  of  these  approaches,  or  combination  of  approaches,  is  best  at  reducing  SD?  Because  the 
potential  of  each  individual  technology  to  reduce  SD  has  not  been  established  (theoretically  or 
empirically),  it  is  difficult  to  postulate  the  “best”  combination.  Rather,  it  seems  that  whichever 
displays  are  best  in  a  given  set  of  circumstances  at  helping  pilots  to  recognize  and  recover  from 
that  SD  event,  then  those  are  the  displays  that  should  be  used.  How  do  researchers  know  which 
displays  are  best  for  different  SD  scenarios?  There  is  nothing  in  the  current  literature  to  suggest 
an  answer;  therefore,  it  will  be  difficult  to  devise  the  “best”  combination.  In  fact,  a  “best” 
combination  or  solution  may  not  be  possible  for  the  foreseeable  future,  due  to  individual  pilot 
variability,  mission  susceptibility,  risk  factors,  and  circumstances.  But,  we  can  theoretically  and 
empirically  determine  good  combinations  of  countermeasures  to  help  combat  SD. 

Determining  the  best  approach  for  converting  unrecognized  into  recognized  SD  events 
should  take  a  human  cognition  and  workload  modeling  approach.  When  visual  senses  are 
overloaded  or  deemed  unreliable,  the  audio  channel  may  be  useful.  A  tactile  (vest)  attitude 
display  may  also  help,  but  research  into  tactile  displays  is  in  its  infancy.  A  presently  untested 
sense  for  SD  is  the  sense  of  smell.  Smells  are  very  compelling19  and  may  be  useful  during 
incapacitating  SD  events  to  help  pilots  “snap  out  of  it.”  Such  an  olfactory  system  will  need 
sensors  to  know  that  the  pilot  is  incapacitated,  and  the  smell  used  must  be  penetrating,  harmless, 
and  easily  carried  aboard  aircraft. 

When  the  pilot  does  not  recognize  an  SD  event,  intelligent  aiding  systems  may  help.  For 
example,  audio  recovery  instructions  have  shown  some  early  utility  with  SD  events  in 
simulators,  according  to  experiments  done  at  the  Air  Force  Research  Laboratory  (AFRL).  They 
key  is  for  the  aiding  system  to  recognize  when  SD  is  causing  an  unusual  attitude  or  dangerous 
situation.  The  emphasis  of  such  systems  ought  to  be  on  preventing  the  bad  consequences  of  SD 
events,  since  it  will  be  virtually  impossible  to  prevent  all  SD  events  due  to  the  nonnal 
physiological  response  to  most  events  that  induce  SD. 

Therefore,  the  best  approach  is  a  layered  one  that  seeks  to  enhance  attitude  awareness  and 
combat  SD  via  multiple  techniques:  training  improvements,  operational  awareness,  better 
displays  (visual,  auditory,  tactile,  olfactory),  intelligent  automation,  and  rigorous  post-flight  SD 
event  analyses  to  better  understand  and  characterize  SD  events,  which  should  also  indicate  the 
operational  effectiveness  of  the  various  SD-combating  techniques.  We  briefly  address  each  layer 
in  the  following  subsections. 

Training  can  help  pilots  recognize  the  types  of  circumstances  that  yield  SD  so  that  they  will 
be  vigilant  for  the  same  or  similar  circumstances.16  Training  also  needs  to  be  enhanced  and 
should  include  classroom  discussion,  simulator  and  centrifuge  training,  and  aircraft 
demonstrations.  While  there  is  some  risk  in  actual  flight  demonstrations,  the  risk  of  not  doing  so 
is  reflected  in  the  rates  of  SD  accidents  in  military  and  civil  aviation  over  many  decades.  Clearly, 
not  all  SD  event  categories  can  be  demonstrated,  but  a  significant  cross-section  could  be.6  The 
teaching  of  SD  situation  recognition  is  a  major  contribution  to  prevention.  And,  pilots  who  are 
more  vigilant  for  SD-inducing  circumstances  are  better  equipped  to  avoid  or  to  successfully 
combat  them.  It  is  also  important  to  teach  recovery  techniques  for  the  riskiest  situations  to  reduce 
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the  number  of  incapacitating  SD  events.  The  most  prevalent  factor  in  USAF  SD  incidents  is 

attention  management,-  a  skill  that  can  (and  should)  be  improved  with  training. - 

Pre-briefings  could  reduce  some  SD  event  severity.  Certainly,  pre -briefings  could  increase 

the  odds  of  any  SD  event  being  recognized  rather  than  unrecognized.  Since  the  environmental 

conditions  that  increase  susceptibility  to  SD  are  known,  mission  pre-briefings  will  be  timely 

reminders  to  pilots,  thus  reducing  the  likelihood  of  an  SD  event  being  unrecognized.  The  Indian 

Air  Force  has  developed  a  checklist  enumerating  SD  threats,  and  has  even  anticipated  the 

increased  threat  of  SD  from  supermaneuvers  in  such  aircraft  as  the  F-22,  F-35,  V-22,  and  the 
2 1 

Apache  helicopter. 

US  Air  Force  statistics  about  contributing  factors  to  SD  incidents  point  to  conditions  in 
which  the  pilot  is  highly  susceptible  to  SD.  All  of  this  knowledge  and  experience  should  be 
highlighted  during  mission  planning  and  pre-briefings  so  that  pilots  can  be  vigilant.  Weather 
conditions,  moonless  night  flights  over  the  desert,  and  night  vision  goggle  (NVG)  use,  are  all 
known  to  be  risk  factors.22  Reminding  pilots  of  the  relevant  mission  risk  factors  and  what  to  do  if 
SD  occurs,  refreshes  their  earlier  training,  and  is  another  important  layer  in  a  comprehensive 
approach  to  mitigating  the  effects  of  SD. 

Because  the  primary  source  of  information  to  the  pilot  is  visual,  because  the  visual  modality 

is  the  most  compatible  modality  for  presenting 
spatial  information  relevant  to  attitude,  and 
because  SD  incidents  are  most  prevalent  in  poor 
weather  or  at  night  (when  the  view  of  the  true 
horizon  is  obscured  or  gone),  it  is  not  surprising 
that  a  key  countermeasure  in  combating  SD  is  in 
the  design  of  effective  visual  displays.  Sadly, 
many  advances  in  this  direction  are  lacking, 
because  design  has  generally  focused  on 
providing  the  necessary  information  to  support 
spatial  orientation,  but  not  on  the  ideal  format  in 
which  that  information  should  be  depicted  most 
naturally  or  intuitively.  One  of  the  major  culprits 
appears  to  be  the  focus  on  using  the  round  dial 
display,  even  as  cockpit  displays  have  evolved  to  include  computer  generated  imagery,  where  the 
electromechanical  constraints  associated  with  the  round  dial  are  no  longer  relevant.  Attitude 
displays  (head-down,  head-up,  and  helmet-mounted)  have  symbology  to  help  recognize  unusual 
or  undesirable  attitudes  that  usually  result  from  SD  (cf.,  Figure  1).  Such  symbology  should  be 

enhanced  based  upon  the  latest  research  results.24'25  In  addition,  how  to  accommodate  the  visual 

22 

limitations  caused  by  NVGs  is  ripe  for  further  research.- 

Recent  results  -  suggest  the  development 
and  testing  of  a  display  icon  that  uniquely 
conveys  the  existence  of  an  SD-induced  unusual 
attitude  and  what  to  do  about  it.  We  propose  that 
this  icon  confonn  to  characteristics  that  make  it 
easy  to  detect,  interpret,  and  act  upon,  without 
confusion  or  control  reversal  errors  (CREs). 

Such  a  display  icon  should  be  a  unique  simple 
geometric  shape  that  portrays  the  relevant 


Figure  2.  SD  icon  snapshots  (piteh/bank):  (a) 
0°/0°;  (b)  +90°(up)/0°;  (c)  0°/+45°(right);  and,  (d) 
+45°/-45. 


Figure  1.  US  military  standard  head-up  display 
(HUD)  symbology. 
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26  ...  .  .  26 
information  in  an  easily  understood  format.  Using  the  principles  espoused  by  Bainbridge,  an 

SD  Icon  should  portray  pitch  and  bank  in  a  way  that  makes  it  clear  what  the  attitude  is,  and  what 

pilots  must  do  to  recover  from  unusual  attitudes.  Figure  2  illustrates  our  concept  for  an  icon  to  be 

presented  on  visual  displays. 

Advances  in  panoramic  displays,  symbology,  and  new  display  types  do  not  answer  the 
question  of  how  to  best  use  each  cockpit  display  to  enhance  pilot  attitude  awareness  and  to 
combat  SD.  Ercoline,  a  leading  SD  researcher,  has  challenged  the  need  for  helmet-mounted 
display  (HMD)  attitude  symbology,  and  has  suggested  a  study  using  four  experimental 
objectives:  agree  on  symbology  across  the  US  military  services;  agree  on  flight  tasks;  develop  a 
protocol;  and,  establish  research  facilities  and  staff.  The  point  is  that  there  is  not  an  expert 
consensus  about  whether  or  how  to  use  HMD  symbology  to  prevent  SD  events  from  becoming 
mishaps. 

Pilots  maintain  attitude  awareness  via  their  visual  scan;  but,  when  SD  occurs,  the  typical 
feeling  is  of  a  conflict  between  their  vision  and  somatic  senses.  It  feels  like  their  vision  is  lying 
to  them.  Therefore,  relying  primarily  on  visual  display  cues  to  recover  attitude  awareness  is 
probably  less  effective  than  using  a  combination  of  multisensory  displays.  Recent  research  has 
explored  using  auditory  and  tactile  senses  to  combat  SD.  This  recent  research  is  laudable 
precisely  because  SD  accident  rates  have  remained  constant  during  the  decades  when  attitude 
information  appeared  only  on  visual  displays.  Clearly,  a  new  emphasis  is  needed.  We  next 
examine  auditory,  then  tactile  countenneasures  for  SD. 

A  key  feature  of  visual  displays  is  their  ability  to  capitalize  upon  parallel  processing,  or 
separate  resources,  so  that,  for  example,  ambient  vision  could  process  a  peripheral  horizon,  even 
as  focal  vision  was  involved  with  other  aspects  of  visual  perception.-  ’  A  long  history  of 
research  on  multiple  processing  resources,10  has  revealed  that  the  auditory  modality  can  process 
information  in  parallel  with  the  visual,  and  therefore  the  auditory  channel  should  be  able  to 
support  spatial  orientation  in  an  otherwise  visually  loaded,  or  visually  impaired  (e.g.,  by 
nystagmus)  environment.  In  making  this  suggestion  however,  it  should  be  noted  that  the  auditory 
modality  is  inherently  less  compatible  for  providing  spatial  information,  than  is  the  visual.23 

Potentially  useful  auditory  information  can  be  classified  into  four  categories;  the  first  two  are 
discrete  or  symbolic,  and  the  second  two  are  continuous  or  analog: 

1 .  Discrete  alerts.  This  describes  the  standard  advantage  of  the  omni-directional  and  intrusive 
auditory  modality  for  presenting  critical  warnings.  For  example,  a  unique  tone  or  single  word 
can  warn  pilots  that  they  may  be  experiencing  SD. 

2.  Verbal  commands.  Verbal  commands  can  provide  effective  corrections  about  what  to  do 

3 1 

to  recover  from  an  SD  incident,  as  the  verbal  modality  is  well  suited  for  command  displays, 
and  there  are  well-documented  successes  in  aviation  for  audio  commands,  such  as  the  “climb 
climb”  resolution  advisory  in  the  Traffic  Collision  Avoidance  System,  or  the  “pull  up” 
command  of  the  Ground  Proximity  Warning  System.  In  both  cases,  the  commands  directly 
signal  an  action,  and  the  verbal  modality  can  successfully  “break  into”  a  visually  saturated 
processing  stream.  It  is  easy  to  envision  how  a  simple  verbal  command  of  the  correction 
required  to  pull  out  of  an  unwanted  attitude  could  be  effective,  (e.g.,  “bank  left”),  assuming 
that  such  a  command  was  offered  following  the  correct  inference  of  aircraft  state. 

3.  Continuous  localization.  A  fairly  extensive  body  of  research  has  established  the  ability  of 
auditory  tones  or  sounds  to  convey  information  as  to  the  location  of  their  source,  by 
mimicking  the  central  and  peripheral  acoustic  effects  of  the  two  ears.33'34  Such  systems 
typically  employ  intensity  and  phase  differences  between  the  two  dichotically  presented 
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sounds,  to  accurately  signal  the  azimuth  of  a  perceived  sound,  relative  to  the  momentary 
orientation  of  the  head.  A  very  naturalistic  or  ecological  signal  of  head  orientation  relative  to 
the  source  of  sound  can  be  provided,  offering  good  support  in  airborne  target  acquisition 
studies.  '  "  However,  an  unfortunate  aspect  of  such  systems  is  that  they  are  far  less  accurate 
in  conveying  elevation  information  than  azimuth  information.  Since  SD  is  often  due  to  a 
misperception  of  which  direction  is  up  rather  than  which  direction  is  forward,  auditory 
localization  may  be  ineffective  as  an  SD  countermeasure. 

4.  Continuous  orientation.  To  provide  a  continuous  signal  of  which  way  is  up  typically 
requires  some  non-naturalistic  assignment  of  head  orientation  to  tone  pitch  and  or  intensity. 
In  this  regard,  effective  demonstrations  of  auditory  compensatory  tracking  have  been 
provided  by  various  researchers.  '  Typical  is  the  assignment  made  by  Vinje  and  Pitkin  in 
which  the  ear  in  which  a  tone  was  presented  represented  the  side  of  an  error  (in  this  case,  an 
attitude  displacement  from  level),  while  the  pitch  of  the  tone  represented  the  magnitude  of 
the  error.  Such  a  system  would  create  an  increasing  pitch  tone,  as  the  aircraft  rolls  to  one  side 
or  the  other.  It  has  been  found  fairly  effective  in  supporting  continuous  compensatory 
tracking,  although  greater  benefits  are  often  realized  when  the  auditory  cues  are  redundant 
with  visual  cues. 

An  important  feature  in  designing  such  auditory  systems  for  SD  applications  is  that  the  pilot 
can  gain  greater  information  from  the  time-varying  trend  of  tones  and  sounds,  than  from  the 
absolute  pitch  or  intensity  levels  of  such  sounds.  For  example,  an  increasing  pitch  (or  volume) 
can  signal  that  things  are  getting  worse.  But  there  is  no  particular  level  of  pitch  (or  volume)  that 
can  be  used  to  intuitively  convey  a  particular  state  (e.g.,  middle  C,  or  70  decibels  corresponds  to 
45°  of  bank).  A  second  issue  is  whether  the  direction  of  a  tone  (e.g.,  left  or  right  ear)  should 
correspond  to  the  status  of  error,  or  the  necessary  command  to  correct  the  error.  Will  a  rising 
tone  in  the  right  ear  intuitively  signal  that  the  right  wing  is  rising  (and  therefore  the  plane  is 
banking  to  the  left),  or  the  converse?  Such  a  mapping  of  tone  to  state  must  be  done  consistently 
and  intuitively  by  the  pilot,  otherwise  control  reversals  will  result  (as  has  been  observed  for  the 
visual  moving  horizon  display  being  confusing).  The  importance  of  this  intuitive  mapping  is 
heightened  for  the  SD  application,  because  the  usage  of  any  auditory  cuing  will  be  required  when 
the  pilot  may  already  be  in  a  stressed  state,  in  which  non-intuitive  mappings  are  likely  to  be 
ignored  or  misinterpreted.40 

In  summary,  audio  tones  are  good  as  alerts  of  undesirable  states,  and  audio  commands  are 
effective  in  directing  recoveries  from  unusual  attitudes;  but  tones  are  not  intuitive  for  orientation 
cueing. 

Early  research  into  tactile  displays  was  by  Gilson  and  colleagues41  who  described  a  lever 
system  attached  to  the  control  yoke.  Their  goal  was  to  indicate  angle  of  attack  (AO  A)  to  pilots 
via  a  lever  that  would  push  into  the  pilot’s  hands  to  indicate  which  direction  the  pilot  should 
move  the  yoke  (push  or  pull)  for  the  optimal  AOA  during  low  speed  flight  phases  (i.e.,  takeoff, 
approach,  and  landing).  Solomonow  and  colleagues42’43  examined  human  sensitivity  to 
vibrotactile  displays,  the  early  forerunners  of  today’s  tactile  vests.  Their  goal  was  to  detennine 
appropriate  vibration  frequency  ranges,  pulse  widths,  and  body  locations  for  maximum 
sensitivity  and  information  transfer.  Kaczmarek  examined  adaptation  and  stimulus  threshold 
changes  with  repeated  vibrotactile  stimuli.44 

Recent  developments  include  full  tactile  vests,  used  to  transfer  altitude  deviation  and  attitude 
information.12,45'50  Tactile  systems  are  typically  used  to  complement  visual  or  audio  displays,  not 
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in  isolation,  because  of  their  lack  of  comparable  sensitivity.  Research  into  this  promising  display 
technology  is  ongoing  due  to  its  relative  ease  of  implementation,  and  due  to  positive  early 
results. 

In  many  respects  the  creation  of  tactile  displays  can  match  applications  1,3,  and  4  of  auditory 
displays  discussed  above,  and  there  is,  indeed,  a  fairly  close  correspondence  between  the  two 
modalities.  Tactile  stimulation  can  (1)  alert,51  it  can  (3)  convey  azimuth  location  information  (for 
example,  a  stimulus  presented  from  a  belt  around  the  waist),  and  it  can  (4)  convey  a  sense  of 
orientation  (e.g.,  differential  elevation  between  the  left  and  right  sides).  Research  on  tactile 
channels  for  orientation  however  is  considerably  sparser  than  for  auditory  channels.45,49  Both 
auditory  and  tactile  modalities  would  probably  suffer  from  habituation,  and  so  must  be  used 
sparingly  and  in  concert  with  other  sensory  inputs. 

Visual  displays  can  convey  precise  spatial  information.  However  both  auditory  and  tactile 
displays  offer  the  potential  to  combat  SD  (or  maintain  spatial  orientation)  because  of  their 
capability  to  provide  infonnation  in  parallel  with  an  overloaded  visual  system.10  Indeed  auditory 
localization  displays  have  provided  useful  spatial  information  in  orienting  toward  targets  in 
three-dimensional  (3D)  space,  ’  as  well  as  in  continuous  tracking.  '  ’  More  recently,  tactile 
and  haptic  displays  have  been  explored  for  similar  purposes.46'49  While  it  is  clear  that 
information  from  these  displays  can  be  processed  in  parallel  with  visual  information,  thereby 
potentially  exploiting  multiple  resources,  it  remains  to  be  seen  the  extent  to  which  such  parallel 
processing  will  take  place  in  the  high  stress  and  workload  of  spatial  disorientation. 

New  technologies,  such  as  3D  audio  and  tactile  displays  can  enhance  recognition  by  using 
non-visual  channels  of  cognition  to  “get  through  to”  the  unaware  pilot.  Since  SD  is  often 
triggered  by  a  disparity  between  visual  and  vestibular  senses,  it  is  wise  to  stimulate  other  senses 
to  aid  recognition.  Audio  is  an  especially  compelling  input  because  it  tends  to  override  other 
senses  for  attention.10  In  addition,  we  can  also  consider  verbal  auditory  information,  such  as 
commands  to  the  pilot  on  how  a  control  should  be  moved,  or  what  instruments  should  be 
consulted,  in  order  to  restore  spatial  orientation.  These  too  are  capable  of  exploiting  the  multiple 
perceptual  resources  of  a  visually  overloaded  (or  incapacitated)  pilot. 

Visual  and  tactile  display  enhancements  may  be  less  effective  during  G-induced  SD  because 
of  vision  tunneling  and  somatic  G-force  effects.47  Researchers  must  also  consider  how  to 
integrate  new  display  ideas  with  existing  cockpit  displays,  audio,  etc.  The  goal  is  to  use  the 
“best”  sensory  cues  for  the  particular  SD  conditions,  if  the  event  characteristics  can  be 
confidently  established. 

In  summary,  it  would  appear  that  a  number  of  features  in  both  visual  and  non-visual  displays 
can  support  the  pilots’  perception  of  attitude.  For  visual  displays,  many  of  these  capitalize  upon 
ambient  vision  and  the  presentation  of  attitude  across  a  wide  range  of  visual  angles.  For  non¬ 
visual  displays,  they  exploit  the  multiple  sensory  resources  of  the  pilot.  Given  other  findings  in 
the  research  literature  supporting  the  advantage  of  redundant  presentations  of  information,31  it 
would  appear  that  offering  attitude  infonnation  through  redundant  channels  could  be  a  valuable 
tool  to  restoring  spatial  orientation.  In  the  following  section,  we  suggest  ways  in  which 
redundant  orientation  information  can  be  intelligently  accomplished. 

Intelligent  onboard  systems  that  can  recognize  SD  and  its  causes  would  provide  an  advantage 
in  the  fight  against  negative  SD  consequences.  Such  systems  would  not  only  tailor  displays  to  the 
particular  SD  event,  but  could  trigger  recovery  actions  via  synthetic  speech  to  talk  the  pilot 
through  the  appropriate  recovery.  Such  techniques  have  shown  utility  in  AFRL  simulator 
experiments,  and  pilots  have  favorable  opinions  of  the  technology.  An  intelligent  system  could 
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also  initiate  recovery  without  jeopardizing  the  mission  by  considering  such  factors  as  position 
relative  to  enemy  airspace,  threat  warning  system  inputs,  and  fire  control  modes.  Triggering  an 
auto-recovery  without  considering  mission  impact  and  pilot  intent  is  unacceptable,  except  in  the 
most  extreme  circumstances.  Auto-recovery  exists  on  the  Swedish  Grippen  in  the  form  of  auto- 
GCAS.  "  However,  US  pilots  distrust  auto-recovery  systems  due  to  potential  errors  and  the 
possibility  of  adversely  affecting  the  mission.  Because  of  these  drawbacks,  a  pilot-activated 
system  could  be  an  interim  step  between  the  present  lack  of  auto-recovery  systems  in  US  aircraft 
and  fully-automated  ones.  However,  a  pilot- activated  recovery  system  would  not  be  effective  for 
unrecognized  or  incapacitating  SD.  Manual  activation  could  be  effective  for  recognized  SD 
events  and  so  should  be  a  part  of  the  layered  solution,  at  least  until  its  efficacy  can  be  proved  or 
disproved. 

Extensive  research  has  been  done  on  aircraft  state,  pilot  intent,  and  hazard  monitoring 
intelligent  systems.  Such  systems  would  be  relatively  easier  to  implement  for  civilian  than  for 
military  aircraft,  and  so  can  be  developed,  tested,  and  certified  in  the  more  benign  civil 
environment  as  a  prelude  to  military  systems,  if  warranted.  An  intelligent  cockpit  system  could 
trigger  the  SD  countenneasure(s)  detennined  to  be  most  effective  in  any  given  situation.  Due  to 
the  likelihood  of  sensory  overload,  an  intelligent  SOAS  would  apply  multisensory 
countenneasures  for  redundancy,  urgency,  and  for  improving  the  odds  of  “getting  through  to” 
the  pilot.  We  have  already  discussed  visual,  auditory,  and  tactile  displays;  one  other  sensory 
channel  remains:  olfactory.  An  intelligent  system  could  trigger  a  unique  harmless  odor  (similar 
to  a  unique  audio  tone)  that  means,  “Spatial  Disorientation  strongly  suspected;  check  your 
instruments!”  For  incapacitated  or  unconscious  pilots,  an  odor  similar  to  smelling  salts  could 
help  the  pilot  regain  consciousness  and  control. 

All  the  old  and  new  technologies  to  combat  SD  incidents  have  uncertain  effectiveness,  unless 
a  thorough  data  gathering  and  analysis  effort  is  begun.  The  goal  should  be  to  better  understand 
and  characterize  SD  events,  and  to  determine  the  efficacy  of  the  various  SD  mitigation  tools. 
Only  with  post  hoc  analyses  will  researchers  have  an  objective  method  to  assess  the  most 
effective  spatial  orientation  enhancing,  or  SD  prevention  and  recovery,  techniques.14  Also,  an 
operational  analysis  system  provides  the  feedback  and  statistics  needed  to  verify  and  validate 
laboratory  research  results.  Lastly,  data  analyses  provide  an  objective  foundation  for  developing 
and  testing  human  performance  models,  such  as  the  multisensory  models  described  below. 

Recent  post-flight  data  analysis  efforts  in  worldwide  commercial  aviation  have  paid  huge 
dividends  in  understanding  the  impact  of  changes  to  such  key  flight  safety  components  as 
airplane  systems,  pilot  training,  air  traffic  control  procedures,  and  airport  conditions.  The  US  Air 
Force  has  recently  begun  such  a  flight  data  monitoring  program  for  its  C-17s.  The  value  of 
regular  consistent  objective  feedback  is  enormous,  as  global  aviation  data  analysis  and  sharing 
programs  attest.  An  analogous  effort  for  SD  research  is  absolutely  vital. 

There  are  many  known  SD  illusions  and  causes.  How  to  best  combat  each  one  is  the 
challenge  of  attitude  awareness  and  SD  researchers.  The  best  solution  is  to  blend  all  of  the 
known  helpful  training,  technologies,  and  techniques,  and  to  tailor  each  to  the  particular  SD 
circumstances.  Researchers  are  investigating  a  wide  variety  of  technologies.  Visual  components 
include  head-down,  head-up,  and  helmet-mounted  displays.  Audio  components  include  3D  audio 
tones  and  verbal  recovery  commands.  Tactile  components  include  vibrating  vests.  Olfactory  cues 
might  be  useful  when  the  pilots’  other  senses  are  overloaded,  or  when  pilots  are  incapacitated. 
Training  includes  classroom  lectures,  centrifuge  demonstrations,  and  practice  flights.  Post-flight 
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data  analyses  provide  feedback  about  which  techniques  are  the  most  effective  under  the  widely 
varying  conditions  of  military  aviation. 


Modeling 

The  goal  of  modeling  attitude  awareness,  spatial  disorientation,  and  multisensory  workload  is 
to  provide  a  framework  for  assessing  or  inferring,  in  real  time,  the  pilot’s  level  of  awareness  of 
the  current  aircraft  orientation,  and  then,  based  upon  this  assessment,  implementing  an 
appropriate  set  of  interventions  to  counter  any  assessed  spatial  disorientation  events  and  their 
consequences.  Because  attention  mismanagement  (i.e.,  distraction)  is  a  key  component  in  many 
SD  mishaps,  we  now  turn  to  modeling  the  pilot’s  attention  in  the  form  of  a  multisensory 
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workload  measure,  SAVVOY,  that  expands  upon  Wickens’  Multiple  Resource  Theory. 

Wickens’  multiple  resource  theory  (MRT)  says  to  use  the  maximum  number  of  different 
sensory  resources  or  channels  to  “get  through”  any  distractions.  Since  our  goal  is  a  multisensory 
approach  to  combating  SD  and  since  Wickens’  MRT  focuses  on  visual  and  auditory  channels, 
we  have  expanded  MRT  to  include  somatic,  vestibular,  and  olfactory  senses.  The  resulting 
workload  model,  which  we  call  SAVVOY  (somatic,  auditory,  visual,  vestibular,  olfactory, 
psychomotor-primary,  psychomotor-secondary,  and  cognitive),  is  useful  for  helping  our  Spatial 
Orientation  Aiding  System  (described  later)  detennine  which  countermeasures  to  apply  under 
varying  conditions.  For  example,  if  the  pilot  is  in  a  high  G  maneuver,  tactile  cues  are  less  likely 
to  be  detected  by  the  pilot  due  to  “high  somatic  workload.”  Similarly,  if  the  pilot  is  talking  on  the 
radio,  audio  cues  may  be  ineffective  due  to  high  audio  workload.  For  psychomotor,  we  have 
primary  and  secondary  observable  actions.  Primary  actions  are  the  pilot’s  flight  control  actions. 
Secondary  actions  are  everything  else  (such  as  pushing  buttons  or  turning  knobs).  We  also  infer 
cognitive  effort  to  complete  the  model  of  pilot  workload. 

Two  models  of  SD  were  developed  to  detect  the  occurrence  of  the  Leans  and  Coriolis 
illusions  -  one  model  for  each  illusion.  Each  model  uses  observable  data  from  the  aircraft  and 
pilot  to  assess  the  state  of  the  vestibular  system  relative  to  the  actual  attitude  of  the  aircraft. 

The  model  of  the  Leans  is  expressed  as  a  timed  sequence  of  events,  with  the  certainty  of  the 
assessment  of  the  disorientation  increasing  with  each  successive  event,  as  shown  in  Figure  3.  The 

first  event  is  the  initiation  of  a  roll 
at  a  rate  below  the  vestibular 
threshold  (Mulder’s  constant  of 
2°/sec).  The  second  event  is  a  roll 
angle  of  greater  than  5°  that  lasts 
longer  than  5  seconds.  If  these  two 
events  occur  in  sequence,  it  is 
possible  that  the  pilot  has  not 
noticed  the  ensuing  roll  angle  and 
that  there  is  a  difference  between 
the  pilot’s  perceived  attitude  and 
the  true  attitude  of  the  aircraft.  As 
such,  the  model  indicates  a 
possibility  of  SD  but  only  at  a  very 
low  confidence  level.  The  third  event  is  the  loss  of  altitude  as  measured  by  negative  vertical 
velocity.  If  this  event  follows  the  first  two,  it  is  possible  that  the  pilot  has  also  not  noticed  the 
loss  of  altitude  and  the  model  represents  an  increased  confidence  in  its  assessment  of  the  Leans 


Figure  3.  Leans  model. 
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(shown  as  increasing  certainty  of  SD  in  Figure  3).  The  fourth  event  is  a  roll  well  above  the 
vestibular  threshold  (e.g.,  greater  than  5°/sec  and  with  sufficient  duration)  in  the  opposite 
direction  from  that  of  the  first  event.  If  this  occurs  following  the  first  three  events,  it  is  possible 
that  the  pilot  has  now  noticed  the  roll  angle  and  has  quickly  corrected  back  toward  level  flight. 
When  this  occurs,  the  pilot’s  vestibular  system  will  register  a  roll  in  the  opposite  direction,  again 
resulting  in  a  difference  between  the  perceived  attitude  and  the  actual  attitude  of  the  aircraft.  At 
this  point  the  model  represents  a  high  level  of  SD  certainty.  The  final  event  in  the  model  is  the 
tilt  of  the  pilot’s  head  opposite  the  perceived  roll  angle.  If  this  occurs  following  the  other  four 
events,  it  is  very  likely  that  the  pilot  is  experiencing  the  Leans  and  the  model  represents  an  even 
higher  level  of  SD  certainty. 

Unlike  the  Leans,  the  model  of  the  Coriolis  illusion  is  not  expressed  as  a  sequence  of  events 
but  as  an  instantaneous  occurrence.  The  model  is  expressed  by  the  maneuvering  of  the  aircraft 
relative  to  the  movements  of  the  pilot’s  head  in  the  three  different  planes  (yaw,  pitch,  and  roll). 
The  first  event  is  the  aircraft  maneuvering  such  that  the  rate  of  change  in  any  of  the  three  planes 
is  above  the  vestibular  threshold  (Mulder’s  constant).  If  during  this  maneuvering  the  pilot’s  head 
moves  greater  than  30°  from  center  in  either  of  the  other  two  planes,  then  there  is  the  potential 
for  the  Coriolis  illusion  to  occur.  For  example,  Coriolis  is  likely  to  occur  if  the  aircraft  is  in  an 
above  threshold  roll  during  which  the  pilot’s  head  moves  a  large  amount  (we  used  30°)  in  either 
the  yaw  or  pitch  planes.  Since  the  Coriolis  model  is  based  on  an  instantaneous  occurrence,  there 
is  no  representation  of  SD  certainty  or  confidence  level.  Rather,  the  model  simply  indicates 
whether  or  not  Coriolis  may  be  occurring. 

“All  models  are  wrong,  but  some  are  useful.  ”54 

At  this  stage  of  our  research  and  development,  the  most  important  benefit  from  modeling  is  a 
deeper  and  more  thorough  understanding  of  the  problem  being  modeled.  In  starting  with  a 
theoretical  approach,  we  ensured  that  we  considered  all  of  the  relevant  aspects  of  attitude 
awareness  and  spatial  disorientation.  Going  from  the  theoretical  to  what  we  could  practically 
accomplish  with  realistic  aircraft  and  state  parameters  meant  that  we  could  prove  the  feasibility 
of  our  approach  while  remaining  cognizant  of  enhancements  to  be  made  in  follow-on  research. 

The  importance  of  effectively  combating  SD  along  the  whole  range  of  operational  exposure 
is  absolutely  critical  to  reducing  the  human  and  aircraft  costs  of  this  major  flying  problem  and 
accident  causal  factor.  Clearly,  it  is  much  less  expensive  to  combat  SD  than  to  replace  pilots  and 
aircraft.  The  time  is  ripe  to  apply  prevention  and  countermeasure  strategies  to  SD  via  pilot 
selection,  training,  mission  briefings,  intelligent  cockpit  aiding  systems,  and  rigorous  post-event 
analyses.  The  following  section  describes  how  we  put  all  of  the  previous  knowledge  together 
into  a  prototype  SOAS  for  pilots. 

Spatial  Orientation  Aiding  System  (SOAS)  Prototype 

The  overall  goal  of  the  Spatial  Orientation  Aiding  System  is  to  achieve  a  multisensory 
approach  to  solving  SD  problems.  The  system  integrates  simulated  multisensory  displays  to  help 
pilots  recognize  and  recover  from  SD  situations.  The  SOAS  prototype  demonstrates  what  could 
be  done  in  either  a  simulator  or  target  aircraft  to  alert  pilots  to  an  SD  situation,  and  to  help  them 
recover. 
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SOAS  consists  of  three  components  (Figure  4).  The  first  component  is  the  State  Table.  It 
contains  state  data  for  the  pilot,  aircraft,  and  external  world,  which  it  uses  to  analyze  the  current 
state  of  the  aircraft,  world,  and  pilot’s  level  of  SD  (if  any)  and  multisensory  workload.  There  is 
also  the  Countermeasure  Assessor,  which  uses  the  output  from  the  State  Table’s  assessors  to 
determine  the  most  effective  countermeasures  for  the  specific  SD  situation.  Once  the 


Figure  4.  Spatial  Orientation  Aiding  System  (SOAS)  architecture. 


countermeasures  are  decided  upon,  the  appropriate  display(s)  are  activated.  As  an  SD  situation 
evolves,  the  State  Table’s  assessors  continue  to  assess  the  state  of  the  pilot,  aircraft,  and  world. 
When  the  situation  changes,  it  informs  the  Countenneasure  Assessor,  so  that  the  compensatory 
actions  can  be  changed  accordingly. 

The  State  Table  helps  SOAS  create  a  complete  picture  of  the  pilot’s  current  situation  and 
serves  as  a  communication  object  among  the  various  processes.  To  create  this  picture,  the  system 
must  know  something  about  the  aircraft,  pilot,  and  the  external  factors  (i.e.,  world)  that  are 
influencing  the  pilot’s  decisions  and  physiology.  The  first  piece  of  this  picture  is  the  aircraft  for 
which  raw  data  are  provided  by  the  aircraft  systems  (or  model,  in  the  case  of  a  simulated 
aircraft).  For  instance,  the  aiding  system  uses  pitch,  bank,  and  airspeed  to  analyze  the  motion  and 
position  of  the  aircraft.  These  raw  data  are  also  used  to  derive  parameters  such  as  pitch  rate,  roll 
rate,  and  change  of  airspeed.  Both  raw  and  derived  data  are  stored  in  the  State  Table. 
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The  second  piece  of  the  picture  is  the  pilot.  SOAS  must  assess  the  current  state  of  the  pilot 
before  it  can  detennine  the  extent  to  which  the  pilot  is  disoriented.  The  pilot’s  state  also  helps 

detennine  the  best  compensatory  actions.  For 
instance,  it  is  important  for  the  aiding  system  to  be 
able  to  detennine  whether  or  not  the  pilot  is 
incapacitated.  To  do  this,  the  aiding  tool  could  use 
sensors  to  measure  the  pilot’s  heartbeat  rate,  skin 
temperature,  pupil  dilation  and  other  physiological 
measures  of  stress  and  incapacitation.  These  pieces 
of  raw  physiological  data  along  with  the  derived 
data  are  stored  in  the  Pilot  State  Table  (Table  1). 

The  last  data  set  needed  to  finish  the  picture  of 
the  pilot’s  situation  is  the  state  of  the  external 
world.  These  data  are  important  because  external 
events  can  increase  the  probability  of  spatial 
disorientation,  and  influence  how  pilots  should 
attempt  to  recover  from  their  disorientation. 
Examples  of  raw  data  are  the  presence  of  threats, 
time  of  day,  visibility,  and  latitude/longitude. 
These  data  can  be  used  to  determine  if  the  pilot  is 
in  combat,  whether  it  is  day  or  night,  and  the  type 

of  terrain  over  which  the  pilot  is  flying. 

Since  there  is  no  “horizon  visible”  sensor  on  any  aircraft  of  which  we  are  aware,  we  can  use 
pilot  head  tilt  angle  to  give  our  aiding  system  an  indication  of  pilot  state  and  orientation  with  the 
visible  horizon.  If  the  aircraft  is  banked  and  the  pilot’s  head  position  indicates  an  opto-kinetic 
cervical  reflex  (OKCR),  then  we  can  assume  that  the  horizon  is  visible.  A  lack  of  OKCR  may 
indicate  that  there  is  no  visible  horizon.  In  any  event,  pilot  head  position  is  important  to  our 
modeling  and  countenneasures  aiding  because  it  is  part  of  the  assessment  of  pilot  susceptibility 
to  SD  (especially  Coriolis). 

For  our  proof-of-concept  demonstration,  the  simulation  was  used  in  a  deterministic  mode  in 
which  the  time  periods  and  state  values  for  each  event  did  not  vary  across  different  model 
executions.  This  allowed  us  to  repeatedly  simulate  exactly  the  same  disorientation  sequence  in 
order  to  evaluate  the  proper  execution  of  the  rest  of  SOAS.  The  simulation  could  also  be 
executed  stochastically  such  that  the  time  and  state  values  would  vary  pseudo-randomly  within 
some  distribution.  This  would  allow  the  model  to  represent  variations  to  the  original  scenarios  to 
help  detennine  the  sensitivity  and  accuracy  of  our  SD  illusion  models. 

The  model  also  includes  SAVVOY  workload  score  calculations.  These  workload  scores  are 
stored  in  the  State  Table  and  used  by  the  Countermeasure  Assessor  to  determine  the  potential 
effectiveness  of  different  compensatory  actions  in  a  specific  SD  situation. 

Once  the  State  Table  is  filled,  SOAS  uses  this  information  to  detennine  the  extent  to  which 
the  pilot  is  experiencing  spatial  disorientation  and  the  best  way  to  help  the  pilot  recover.  The 
assessors,  which  are  part  of  the  State  Table,  do  the  first  part  of  this  task.  They  determine  the 
extent  to  which  the  pilot  is  spatially  disoriented  and  the  risk  sunounding  the  current  situation. 
These  different  assessors  within  the  State  Table  will  be  refened  to  as  the  State  Assessor  in  the 
rest  of  this  description  of  the  SOAS  prototype.  The  Countermeasure  Assessor  determines  the 
appropriate  compensatory  actions,  based  on  the  outputs  of  the  State  Assessor. 


Table  1.  Pilot  state  parameters 


Parameter 

Range 

Head  position  pitch 

-90-+90 

Head  position  roll 

-90-+90 

Head  position  yaw 

-120- 

+120 

Somatic  workload 

0-7 

Auditory  workload 

0-7 

Vestibular  workload 

0-7 

Visual  workload 

0-7 

Olfactory  workload 

0-7 

Psychomotor 
primary  workload 

0-7 

Psychomotor 
secondary  workload 

0-7 

Cognitive  workload 

0-7 
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Since  the  Leans  model  is  based  on  a  sequence  of  SD  triggering  events,  the  State  Assessor 
keeps  track  of  which  events  have  occurred  within  the  sequence.  In  addition,  it  must  determine  if 
trigger  events  that  have  already  occurred  are  still  valid.  Each  time  the  State  Table  is  updated,  the 
Leans  Assessor  begins  by  looking  for  a  roll  rate  that  the  vestibular  system  does  not  notice  using 
the  axis  acceleration  multiplied  by  its  duration  being  below  Mulder’s  constant.  For  the  next  State 
Table  update,  it  first  checks  that  any  maneuvers  have  exceeded  Mulder’s  constant.  If  Mulder’s 
constant  is  exceeded  (i.e.,  the  motion  is  detected),  the  Leans  Assessor  resets  the  first  trigger  and 
starts  over  again.  But,  if  Mulder’s  is  still  not  exceeded,  the  Leans  Assessor  then  looks  for  a  roll 
angle  of  greater  than  5  degrees.  As  the  scenario  continues,  if  the  pilot’s  vestibular  system  is  still 
not  reacting  based  on  the  Mulder’s  calculation  and  the  roll  angle  stays  at  greater  than  5  degrees 
for  greater  than  5  seconds,  the  assessor  will  detennine  that  a  Leans  SD  has  begun  and  will  inform 
the  rest  of  the  system  that  a  Leans  is  occurring  with  a  low  level  of  confidence.  If  these  conditions 
continue,  the  assessor  will  check  to  see  if  the  aircraft  is  losing  altitude.  If  it  is,  then  the  assessor 
will  increase  the  confidence  level  to  medium.  If  these  conditions  continue,  the  assessor  will  look 
for  an  opposite  roll  above  the  threshold  of  the  vestibular  system.  If  this  occurs,  the  assessor  will 
increase  the  SD  confidence  level  to  high.  Finally,  if  all  these  conditions  have  occurred  in 
sequence  and  the  pilot’s  head  tilts  in  the  opposite  direction  from  the  roll  back,  the  SD  confidence 
level  is  increased  to  very  high. 

When  an  SD  event  has  been  detected,  the  system  must  detennine  the  severity  of  the  pilot’s 
situation.  The  importance  of  SD  severity  can  be  seen  in  the  SD  remediation  network  (Figure  5). 

The  network  implies 
that  an  increasing 
severity  of  SD  warrants 
an  increasing  level  of 
aiding.  This  is  where 
the  other  components 
of  the  State  Assessor 
become  important.  The 
Aircraft  and  World 
Assessors  help  the 
State  Assessor  estimate 
the  severity  of  the 
pilot’s  situation  by 

analyzing  aircraft  and  world  state  data.  SD  events  can  be  divided  into  three  primary  levels  of 
severity,  where  each  level  infers  the  goal  of  SO  AS. 


SD  possible 


SD  event 
detected 


Defining  conditions:  Defining  conditions: 

-  Aircraft  state  -  Aircraft  state 

-  External  conditions  -  External  conditions 

-  Pilot  state  -  Pilot  state 


Types  of  aiding 


Onset  Situations 


Types  of  aiding 


►  Evolving  Situations 


Defining  conditions: 

-  Aircraft  state 

-  External  conditions 

-  Pilot  state 


Types  of  aiding 


►  Critical  Situations 


Recovery 


SD  event 
terminated 


Increasing  severity  of  SD 


Figure  5.  SD  remediation  network. 


These  levels  are: 

1 .  Onset  situations:  Emphasis  on  prevention  (Figure  6) 

2.  Evolving  situations:  Emphasis  on  helping  the  pilot  recognize  and  recover 

3.  Critical  situations:  Emphasis  on  recovery 
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The  Aircraft 
Assessor 
monitors  the 

aircraft’s  state. 
Its  primary  goal 
is  to  estimate 
the  situation’s 
current  risk 

based  on  the 
aircraft’s 
acceleration 
and  position. 
For  instance, 
the  pilot  is  in  a 

more  severe  situation  if  the  aircraft  is  in  an  unusual  attitude  and  about  to  crash,  and  this  assessor 
monitors  the  air-craft’s  time  until  impact  with  the  terrain.  The  less  time  the  pilot  has  to  recognize 
and  recover  from  SD,  the  more  intrusive  the  countenneasures  must  be. 

The  World  Assessor  monitors  the  external  world’s  state.  In  particular,  it  is  concerned  with 
factors  that  contribute  to  or  increase  the  pilot’s  vulnerability  to  SD.  These  factors  include 
visibility  (Is  the  horizon  visible?),  and  the  presence  of  enemy  threats.  One  example  is  that  pilots 
are  more  susceptible  to  SD  if  they  cannot  see  the  real  horizon.  Another  example  is  that  we  may 
suspend  counter-measure  actions  if  the  pilot  is  jinking  to  avoid  an  enemy  missile. 

These  types  of  calculations  are  difficult  to  do  because  the  required  parameters  are  hard  to 
measure  with  the  technology  that  is  currently  available  in  an  aircraft.  We  do  not  perfonn 
calculations  in  the  SOAS  prototype  that  we  are  not  able  to  do  with  data  provided  by  a  real 
simulator  or  aircraft.  Consequently,  the  analyses  that  occur  in  this  portion  of  State  Assessor  are 
conservative.  These  concepts  are  included  in  our  design  to  ensure  completeness.  The 
Countermeasure  Assessor  uses  the  SD  type  and  level  from  the  State  Assessor  to  determine  what 
type  of  cues  are  appropriate  countermeasures.  It  also  uses  SAVVOY  workload  levels  from  the 
State  Table  to  detennine  the  predicted  effectiveness  of  each  countermeasure.  Once  the 
Countermeasure  Assessor  has  selected  appropriate  countermeasures  for  the  SD  situation,  it  sends 
this  information  to  the  (simulated)  aircraft  displays.  The  State  Table  also  stores  which 
countermeasures  are  presently  being  used,  so  that  the  State  Assessor  knows  which  pilot  resources 
are  being  loaded. 

If  the  State  Assessor  detects  a  change  in  the  current  SD  event,  it  alerts  the  Countenneasure 
Assessor  of  this  change.  For  instance,  if  the  State  Assessor  estimates  that  the  pilot  has  fully 
recovered  from  the  SD  event,  it  passes  this  infonnation  to  the  Countermeasure  Assessor.  Upon 
receiving  this  information,  the  Countermeasure  Assessor  tenninates  the  various  countenneasures 
it  initiated.  The  State  Assessor  also  infonns  the  Countermeasure  Assessor  when  the  SD  situation 
has  either  decreased  or  increased  in  severity,  so  that  the  countermeasures  can  be  changed 
accordingly. 

The  prototype’s  Countermeasures  represent  technologies  currently  available  in  a  cockpit, 
such  as  helmet-mounted  displays,  as  well  as  new  technologies,  such  as  tactile  vests.  The 
Countermeasures  are  represented  in  a  simplified  cockpit  image  (Figure  7)  for  proof-of-concept 
demonstration  purposes. 


Types  of  Aiding: 

-  Normal  displays  (HDD, 

HUD,  HMD,  low  tactile) 

-  No  audio,  olfactory  displays 

Types  of  Aiding: 

-  Enhanced  displays  (HDD, 

HUD,  HMD,  low  tactile) 

-  Low  audio 

-  No  olfactory  display 

Types  of  Aiding: 

-  Enhanced  displays  (HDD, 

HUD,  HMD) 

-  Medium  tactile 

-  Low  audio 

-  No  olfactory  display 

uuuu  nui  izui  i 

uusluicu  nui  uum 

Horizon  Gone  - ► 

Defining  States: 

-  Aircraft:  controlled  flight 

-  External:  good  visibility 

-  Pilot:  no  problems 

Defining  States: 

-  Aircraft:  controlled  flight 

-  External:  fair  visibility 

-  Pilot:  no  problems 

Defining  States: 

-  Aircraft:  controlled  flight 

-  External:  poor  visibility 

"  Pilot:  possibly  distracted 

Figure  6.  SD  Onset  remediation  network. 
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Figure  7.  Demonstration  screen  shows  multisensory  countermeasures  and 

assessor  outputs. 

Our  SOAS  prototype  builds  upon  prior  intelligent  cockpit  systems,  so  we  have  confidence 
that  the  design  is  robust  and  suitable  for  a  range  of  applications,  including,  spatial  disorientation. 
Our  demonstration,  which  includes  the  two  illusions  described  earlier,  plus  a  scenario  based 
upon  actual  data  obtained  from  the  Air  Force  Safety  Center  proves  the  feasibility  of  our  model- 
based  concept  and  approach. 


Conclusions 

For  this  proof-of-concept  research,  we  accomplished  the  following  major  objectives: 

•  We  performed  a  literature  search,  including  Russian  literature,  which  formed  the  foundation 
of  our  quantitative,  model-based  approach. 

•  We  designed  and  developed  models  for  the  Leans  and  Coriolis  illusions.  The  Leans  model 
worked  with  the  actual  F-16  data  that  we  received  from  the  Air  Force  Safety  Center  (AFSC) 
within  the  last  weeks  of  this  project’s  technical  work.  We  are  especially  delighted  that  our 
SOAS  is  robust  enough  to  work  with  this  data  set  in  a  sensible,  realistic  manner.  (For  the 
actual  SD  flight  data,  given  to  us  by  AFSC,  we  took  the  raw  data  parameters  that  we  needed, 
and  added  small  random  numbers  for  other  parameters  not  found  in  that  data  set.  Then,  we 
ran  the  data  through  our  SOAS  looking  for  a  Leans-like  event.  We  did  not  try  our  Coriolis 
model  on  the  actual  data  because  we  did  not  have  pilot  head  position  data.  Even  without 
head  position  data,  our  Leans  model  worked  because  the  5th  step  in  Figure  3,  a  head  tilt,  only 
increases  our  confidence  level;  it  is  not  required  for  a  Leans  assessment.) 
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•  We  enhanced  Wickens’  Multiple  Resource  Theory  to  account  for  the  other  senses  critical  to 
this  effort.  We  called  the  resulting  “theory”  SAVVOY  in  honor  of  Dr.  Wickens’  location  in 
Savoy,  Illinois  (at  the  University  of  Illinois’  Aviation  Research  Laboratory).  The  newly 
included  senses  are  somatic,  vestibular,  and  olfactory.  A  model  of  pilot  workload,  using 
hypothetical  (but  realistic)  SAVVOY  scores,  influenced  the  SD  countermeasures  applied  by 
our  SOAS. 

•  We  invented  an  SD  icon  (Figure  2)  to  be  an  intuitive  representation  of  the  aircraft  pitch  and 
roll.  How  intuitive  it  actually  is  remains  to  be  seen  via  testing  in  ongoing  research. 

•  Most  importantly,  we  delivered  a  working  SOAS  prototype  that  serves  as  an  excellent 
foundation  for  follow-on  research,  development,  and  testing  with  pilots  in  a  motion-based 
simulator,  and  in  flight  tests. 

The  anticipated  benefits  of  this  research  and  development  are  to  help  pilots  maintain  spatial 
orientation  in  flight,  or,  when  orientation  degrades,  to  recognize  and  recover  from  spatially 
disorienting  situations,  thus  preventing  SD’s  negative  consequences.  The  cost  impact  of  SD  on 
the  US  military  is  over  $300  million  per  year,  not  counting  lives  lost.  US  civilian  losses  are 
comparable.  Devising  a  layered  approach  to  even  incrementally  improve  the  situation  has  a 
multi-million  dollar  per  year  potential  benefit. 
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ABSTRACT 

Introduction.  Aviation  spatial  disorientation  mishaps  remain  a  concern,  especially  due 
to  their  fatality  rate.  Some  of  the  most  insidious  disorientations  are  due  to  vestibular  stimuli  in 
the  absence  of  visual  cues.  A  category  of  such  disorientations  are  known  as  somatogyral 
illusions.  Methods.  To  detennine  the  effects  of  spin  rate  and  duration  on  the  perception  of  the 
somatogyral  illusion,  we  examined  the  subjective  response  of  pilots  and  non-pilots  to  rotation 
around  the  yaw  axis  in  a  flight  simulator,  in  a  manner  that  would  mimic  two  vestibular  illusions 
found  in  flight:  the  washout  of  the  semi-circular  canals  following  sustained  turns,  and  the 
illusory  counter-rotation  following  return  to  straight  and  level  flight.  Twenty-nine  subjects  (14 
pilots)  were  seated  blindfolded  in  a  flight  simulator  which  accelerated  to  constant  plateau 
rotation  rates  of  20,  70,  and  120  degrees/sec,  and  then  decelerated  to  stationarity;  plateaus  were 
10,  20  or  40  seconds.  Subjects  reported  (a)  the  time  when  the  perception  of  rotation  ceased  (i.e., 
the  subjective  time  until  washout  was  reached),  (b)  the  relative  magnitude  of  the  counter-rotation 
experienced;  and,  (c)  the  time  until  the  perception  of  counter-rotation  ceased.  Subjects  also 
manipulated  a  slider  to  provide  a  continuous  subjective  measure  of  their  experience  of  rotation. 
Results.  The  two  time  measures  increased  with  increases  in  both  the  duration  and  magnitude  of 
the  spin.  The  increase  in  perceived  washout  time  with  spin  rate  was  non-linear  (geometric). 
There  was  an  interaction  between  spin  duration  and  spin  rate  on  the  experience  of  illusory 
counter-rotation  magnitude,  such  that  at  low  rates,  spin  duration  had  no  effect,  but  its  effect 
increased  at  faster  rates.  The  time  constant  of  adaptation  of  the  semicircular  canals  was 
estimated  to  be  8.3  seconds.  Discussion.  The  effects  were  validated  against  a  model  of 
semicircular  canal  and  cupola  adaptation,  which  predicted  the  data  with  high  accuracy.  Pilots 
and  non-pilots  did  not  differ  in  their  illusory  experience. 

INTRODUCTION 

Spatial  disorientation  (SD)  mishaps  remain  an  enduring  concern  in  aviation.  While  their 
number  in  civilian  (general)  aviation  is  not  large  (1.9%  in  a  recent  tabulation),  over  90%  of  those 
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that  occur  are  fatal  [16].  Furthermore,  their  frequency  is  growing  in  U.S.  military  operations, 
estimated  to  be  between  6  and  21%  (depending  on  branch  of  service)  in  one  study  [8],  and  being 
a  contributing  cause  in  nearly  38%  of  U.S.  Air  Force  aviation-related  mishaps  [13].  The 
continued  incidence  and  seriousness  of  SD  has  prompted  researchers  to  seek  new  methods  to 
both  predict  spatial  disorientation  and  to  counteract  it. 

A  recent  effort  [11]  developed  a  model  that  will  predict  when  a  pilot  is  in  danger  of 
becoming  disoriented.  Aircraft  motions  are  compared  to  a  predictive  model  that  determines  if  a 
pilot  may  experience  several  known  illusions,  including  the  somatogyral  illusion,  the  leans,  and 
the  somatogravic  illusion  [4],  The  first  of  these  illusions  involves  the  semicircular  canals  and 
yaw  rotational  motion  around  the  vertical  axis.  During  a  sustained  turn,  the  rate  of  subjective 
experience  of  rotation  gradually  declines  below  the  rate  of  true  rotation,  a  “washout”  of  the 
canals,  until  eventually  a  point  of  subjective  stationarity  is  reached,  reflecting  the  dynamics  of 
the  cupola  [2,  5].  Then,  when  sustained  rotation  is  halted  (e.g.,  rolling  out  of  a  turn)  and  straight 
flight  is  resumed,  an  illusory  sense  of  counter-rotation  is  perceived,  again  reflecting  cupola 
dynamics.  Here  again  the  sense  of  illusory  rotation  returns  to  zero  after  some  decay  period.  It  is 
this  illusory  counter-rotation  that  causes  the  somatogyral  illusion  and  is  a  major  contributor  to 
the  leans,  and  to  occasional  extension  into  a  more  serious  graveyard  spiral  [4].  Our  model  [11] 
captures  the  joint  impact  of  these  two  illusions  -  washout  and  illusory  counter-rotation  -  as  they 
might  characterize  a  pilot’s  experience  when  returning  to  straight  flight  after  a  sustained  turn. 

For  the  purposes  of  this  study  we  assumed  that  the  perceptual  effect  of  hank  change  when  rolling 
out  of  a  turn  as  insignificant  to  the  overall  effect  of  yaw  axis  rotation  perception.  The  model 
incorporates  assumptions  about  vestibular  thresholds  and  rates  of  washout  at  different  angular 
velocities  (rates  of  turn)  and  their  durations. 

Continued  development  of  the  model  requires  further  research  and  characterization  of  the 
semicircular  canals  at  rotation  rates  that  are  similar  to  those  experienced  in  flight,  with  relatively 
rapid  sustained  turns.  Prior  studies,  while  examining  these  dynamics  in  a  quantitative  fashion  [2, 
5,  9]  have  either  used  vestibular  inputs  quite  different  from  those  characterized  by  sustained 
aircraft  turns  (e.g.,  using  triangular  or  short  inputs)  or  have  focused  only  on  the  illusory  counter¬ 
rotation  phase  [9].  Also,  these  studies  have  not  systematically  mapped  both  initial  washout  and 
illusory  counter-rotation  to  a  range  of  values  characterizing  the  turn:  both  its  rate  and  its 
duration.  This  is  our  goal  in  the  current  research. 

There  are  a  variety  of  means  of  assessing  perceived  rotation  [9].  Psychophysical 
magnitude  estimation  [10,  12]  requires  the  subject  to  offer  a  single  numerical  value.  Analog 
devices  can  be  classified  into  three  general  categories:  (1)  direct  pointing  requires  pointing  to,  or 
verbalizing,  alignment  with  a  fixed  perceived  landmark  (orientation)  in  the  external  world  [3,5, 
14];  (2)  indirect  pointing  via  position  control  (e.g.,  matching  perceived  to  controlled  position,  or 
perceived  to  controlled  velocity)  involves  rotating  a  wheel  or  crank  at  a  velocity  proportional  to 
the  perceived  body  rotation  [7,  9];  and,  (3)  indirect  pointing  via  velocity  control  involves  moving 
a  joystick  or  slider  to  a  position  that  corresponds  to  the  perceived  rotational  velocity  [1,6].  To 
these  explicit  subjective  psychophysical  techniques  can  be  added  an  implicit  biological  measure, 
which  indicate  the  brain’s  sense  of  rotation  by  recording  ocular  nystagmus  [5,9]. 
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While  we  employ  psychophysical  measures  here  to  assess  the  perceived  magnitude  and 
duration  of  rotation,  these  are  insufficient  to  capture  the  continuous  decay  of  perceived  rotation 
related  to  washout.  Direct  pointing  is  awkward  and  difficult  when  rotations  span  beyond  about 
90  degrees  (requiring  pointing  behind  oneself)  [5],  Indirect  position  control  is  plausible,  and 
found  to  correlate  well  with  nystagmus  measures  [9],  but  can  be  fatiguing  if  sustained  for  long 
durations.  Because  data  from  other  tracking  domains  indicates  approximate  equivalence 
between  position  control  and  velocity  control  [15]  and  because  we  wished  to  avoid  possible 
fatigue  effects,  we  chose  the  velocity  control  technique  whereby  slider  position  indicated  the 
perceived  velocity  and  direction  of  rotation. 

The  purpose  of  the  current  experiment  is  to  fully  understand  the  dynamics  of  the  yaw 
semicircular  canal  as  these  are  exercised  in  the  somatogyral  illusion  expressed  through 
perception,  and  experienced  in  the  context  of  a  rotating  aircraft  simulator,  with  dynamics 
characteristic  similar  to  those  that  might  be  experienced  by  pilots  in  flight.  These  dynamics  were 
created  by  a  trapezoidal  spin  profile  consisting  of  a  constant  acceleration  of  rotation 
(corresponding  to  entering  a  turn),  leading  to  a  constant  (plateau)  rotation  rate  (angular  velocity 
or  spin),  set  at  one  of  three  levels  (20,  70,  120  deg/sec.)  and  persisting  for  one  of  three  durations 
(10,  20,  40  sec.),  followed  by  a  constant  deceleration  (corresponding  to  exiting  the  turn  by 
rolling  back  to  straight  and  level  flight),  followed  finally  by  a  stationary  period.  The  three  levels 
of  rotation  speed  were  chosen  to  span  a  range  from  relatively  tight,  but  still  nonnal,  turns  (20 
degrees/sec)  to  rapid  rotation  typical  of  a  graveyard  spiral  (120  deg/sec).  The  three  durations 
were  chosen  to  span  a  range  that  was  short,  but  not  so  short  that  the  transient  acceleration 
response  of  the  cupola  had  not  yet  been  completed  (10  sec),  to  sufficiently  long  (40  sec)  to 
assume  that  washout  will  have  occurred  [5], 

During  the  constant  rotation  plateau,  vestibular  washout  should  be  experienced  as  the 
rotation  rate  of  semicircular  canal  fluid  reaches  the  rotation  of  the  skull  (and  simulator)  at  which 
point  subjective  stationarity  should  be  perceived.  During  the  subsequent  deceleration  and  true 
stationary  period,  illusory  counter-rotation  should  be  experienced  in  a  direction  opposite  from 
the  original  rotation.  The  magnitude  should  be  equal  to  that  of  the  final  rate  of  semicircular 
canal  fluid  rotation  preceding  the  deceleration  phase;  but  this  illusory  rotation  too  should  wash 
out  following  prolonged  steady  state  (0  acceleration)  conditions  as  the  cupola  return  to  their 
steady  state.  Pilots  expressed  their  perception  of  rotation  (whether  true  or  illusory)  by  (a)  sliding 
a  slider  to  correspond  with  the  degree  of  perceived  rotation,  (b)  indicating  vocally  their  degree  of 
experience  of  initial  illusory  rotation  following  deceleration  from  the  constant  rotation,  and  (c) 
indicating  the  point  of  perceived  stationarity  (in  both  true  and  illusory  rotation)  by  both  a  verbal 
and  manual  command. 

In  the  3  (spin  plateau  duration)  x  3  (spin  rate  during  plateau)  experimental  design,  we 
made  the  following  predictions,  with  varying  degrees  of  certainty:  (1)  For  a  given  spin  velocity, 
longer  durations  will  make  it  more  likely  that  washout  will  be  reached,  leading  after  the  spin  is 
halted  to  both  a  longer  duration  illusory  rotation  and  a  higher  perceived  magnitude  of  that 
rotation;  (2)  for  a  given  spin  plateau  duration,  a  faster  spin  will  increase  the  terminal  rotation 
within  the  semicircular  canal  (and  cupola  deflection),  thereby  also  leading  to  a  longer  illusory 
counter-rotation  and  greater  perceived  illusory  magnitude;  and,  (3)  less  certain  is  whether  and 
how  these  two  independent  variables  will  interact.  It  is  possible,  however,  to  predict  that  with  a 
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slower  spin,  the  semicircular  canal  will  be  more  likely  to  have  reached  washout  earlier,  and  so 
lengthening  the  initial  spin  duration  will  have  less  of  an  effect  on  the  two  illusion  variables  (time 
and  perceived  magnitude)  than  its  effect  will  be  with  the  faster  spins.  We  will  examine  these 
predictions  with  an  ANOVA  model.  A  second  way  to  test  our  predictions  is  by  comparing  our 
data  with  the  predictions  made  by  our  vestibular  model  [11],  which  makes  quantitative 
assumptions  regarding  the  strengths  of  washout  and  illusion  buildup,  integrating  these 
implications  for  the  prediction  of  the  dependent  variables. 


Our  goals  in  the  experiment  were  thus:  (a)  to  assess  the  consistency  with  which  the 
known  dynamics  of  the  semicircular  canal  would  be  reflected  by  vestibular  perception  in 
subjects  experiencing  rotation  within  a  simulator;  (b)  to  assess  the  consistency  with  which  the 
dynamics  creating  washout  from  true  rotation  would  also  create  illusory  counter-rotation  of 
identical  or  similar  magnitude;  (c)  to  provide  an  estimate  of  parameters  that  could  be  used  in  our 
model  [11]  for  predicting  spatial  disorientation  in  real  time  and  to  use  the  model  to  predict  the 
obtained  data;  and,  a  final  goal  was  (d)  to  compare  the  extent  to  which  the  observed  patterns  of 
illusions  was  modified  by  flight  experience. 

METHODS 

Twenty-nine  subjects,  all  of  whom  were  cadets  between  18  and  23  years  of  age  at  the 
United  States  Air  Force  Academy,  volunteered  for  the  study.  Fourteen  of  the  subjects  were 
qualified  glider  or  small-motor  aircraft  pilots  with  over  50  hours  logged  flight  time.  The 
remaining  fifteen  had  little  to  no  flight  experience.  The  study  protocol  was  approved  in  advance 
by  the  Air  Force  Academy’s  Institutional  Review  Board.  Each  subject  provided  written 
informed  consent  before  participating. 

All  spins  were  completed  in  a  GAT-II  (Enviromnental  Tectonics  Corporation,  Inc.)  three- 
degree  of  freedom,  motion-based  flight  simulator  with  the  instrument  panels  and  out-the -window 
display  screen  blanked.  Each  subject  experienced  a  total  of  18  different  test  profiles  over  two 
days.  Trapezoidal  spin  profiles  were  utilized  for  the  study:  the  GAT-II  started  at  rest, 
accelerated  at  a  rate  of  10  deg/sec2,  maintained  a  constant  angular  velocity  of  20,  70,  or  120 
deg/sec,  and  then  decelerated  at  a  rate  of  10  deg/sec  .  The  plateau  durations  lasted  10,  20,  or  40 
seconds.  Each  of  the  nine  profiles  were  experienced  in  both  the  clockwise  and  the 
counterclockwise  directions  (3  angular  rates  x  3  plateau  durations  x  2  directions  =18  trials). 

During  testing,  subjects  were  blindfolded  and  placed  their  heads  in  a  stable  headrest 
within  the  simulator  cab.  They  were  asked  to  report  perceived  rotation  rates,  changes  in 
perceived  motion,  and  cessation  of  motion.  A  BIOP  AC®  TS1 15  Variable  Assessment 
Transducer  or  “slider”  was  used  to  continuously  record  perceived  rotation  rate.  The  slider  had  a 
scale  of  0-9,  with  4.5  as  the  starting  “no  rotation  perceived”  point.  Subjects  held  the  analog 
slider  in  their  lap  with  one  hand,  and  controlled  the  slider  with  the  other.  Upon  the  start  of  each 
profile,  subjects  moved  the  slider  in  the  direction  of  perceived  rotation  until  maximum  velocity 
was  perceived.  The  slider  position  remained  stationary  if  the  subjects  perceived  a  constant 
angular  velocity,  and  upon  a  perceived  deceleration,  the  subjects  moved  the  slider  back  towards 
the  middle.  If  a  complete  stop  was  felt  (i.e.,  subjective  stationarity),  they  reported  the  stop 
verbally  and  by  tapping  a  BIOP  AC ®  digital  TSD1 16B  foot  switch.  This  point  was  then  used  as 
the  new  “neutral”  or  zero  position;  subjects  were  not  expected  to  scale  their  perceptions  so  that 
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they  would  return  to  the  4.5  position  on  the  slider.  During  the  deceleration  of  the  GAT,  subjects 
typically  perceived  a  rotation  in  the  direction  opposite  from  their  actual  spin.  To  record  this 
perception,  the  subject  then  moved  the  slider  in  the  direction  opposite  of  their  initial  input  to 
indicate  the  magnitude  of  this  perceived  counter-rotation.  As  before,  when  the  illusory  motion 
diminished,  they  moved  the  slider  back  toward  the  center,  and  when  motion  perception  stopped, 
they  provided  a  verbal  report  of  “stop”  and  tapped  the  foot  switch. 

Verbal  reports  and  the  foot  switch  were  used  concurrently  to  report  two  other  events. 

The  first  report  usually  involved  the  shorter  duration  (10-  and  20-second)  plateaus  when  the 
subjects  often  did  not  experience  washout  and  subjective  stationarity.  During  these  profiles, 
perceived  rotation  could  go  directly  from  a  left  spin  to  a  right  spin  (or  vice  versa).  In  these 
instances,  subjects  were  instructed  to  simply  report  “switch”  and  tap  the  foot  switch.  The  other 
event  that  was  reported  was  the  time  when  the  maximum  perceived  rotation  rate  was  achieved. 

The  maximum  perceived  velocity  was  measured  on  a  subjective  scale  of  1  to  10. 

Subjects  were  instructed  to  assign  a  value  of  10  for  the  magnitude  of  the  maximum  velocity  in 
the  initial  spin  direction  for  each  profile.  Then,  upon  reaching  the  apparent  maximum  velocity  in 
the  alternate  direction  (i.e.,  during  illusory  counter-rotation),  subjects  were  asked  to  quantify  how 
fast  they  perceived  they  were  spinning  relative  to  the  “10”  rating  in  the  initial  direction.  For 
example,  if  they  perceived  they  were  spinning  half  as  fast  after  the  switch  in  direction  as  they 
were  before  the  switch,  they  reported  a  “5.”  A  review  of  data  integrated  by  Guedry  [5]  indicates 
that  with  a  relatively  modest  amount  of  training,  subjects  are  able  to  accurately  reflect  the  true 
rate  of  rotation  with  such  verbal  estimates. 

For  each  subject,  data  collection  took  place  over  two  separate  one-hour  sessions  with  at 
least  24  hours  between  each  session.  The  first  session  began  with  three  training  spins  along  with 
six  experimental  spins;  the  second  session  consisted  of  the  remaining  12  spins.  The  subjects 
rested  for  at  least  60  seconds  between  each  test  run.  The  training  run  profiles  were  20,  70,  and 
120  deg/sec  with  40-second  plateaus  to  familiarize  the  subjects  with  the  different  angular 
velocities,  use  of  the  equipment,  and  the  test  methodology.  The  order  of  the  profiles  was 
randomized  using  a  Latin-square  sequence.  Subjects  encountered  two  quasi  random  sequences 
of  the  nine  profiles.  A  profile  administered  in  one  direction  (e.g.,  clockwise)  in  the  first 
sequence  was  subsequently  encountered  in  the  other  direction  in  the  second  sequence.  Order 
was  further  constrained  so  that  profiles  alternated  directions  to  prevent  the  subject  from 
adaptation  during  prolonged  spinning  in  one  direction.  Thus  the  direction  of  spin  (unlike  the 
magnitude  or  duration)  could  always  be  anticipated.  The  first  sequence  of  9  profiles  was 
distributed  between  day  1  and  day  2,  while  the  second  sequence  was  entirely  administered  on 
day  2.  (However,  because  of  a  procedural  error,  only  half  the  subjects  provided  data  for  the  70 
deg/sec,  20-sec  spin.) 


RESULTS 

A  representative  graph  from  a  70  deg/sec,  40-second  plateau  trial  is  shown  in  Figure  1. 
The  spin  profile,  the  slider  data,  the  verbal  responses,  and  the  foot  switch  inputs  are  all  depicted. 
From  this  graph,  two  major  dependent  variables  were  calculated:  the  time  to  subjective 
stationarity  (TSS)  and  the  duration  of  illusory  motion  (DIM).  As  reported  in  Figure  1,  the  TSS  is 
the  time  from  the  beginning  of  the  plateau  until  the  subjects  felt  they  were  no  longer  spinning  (an 
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event  which  did  not  occur  for  all  profiles,  particularly  the  shorter  ones);  and  the  DIM  is  the  time 
from  the  cessation  of  GAT  motion  until  the  perceived  illusory  self-motion  ceased. 


Figure  1.  A  graph  of  a  counterclockwise  (left)  spin  at  70  degrees/second  for  40  seconds.  Initial  and 
subsequent  max  velocities,  actual  and  perceived  stopping  points,  and  the  switch  in  direction  are  marked  by 
vertical  lines  as  indicated  in  the  legend.  The  TSS  is  measured  from  the  beginning  of  the  actual  profile’s 
plateau  to  the  first  perceived  stop.  DIM,  the  time  from  the  cessation  of  actual  motion  until  the  perception  of 
rotation  in  the  opposite  direction  ends  is  measured  from  the  end  of  the  profile  to  the  second  perceived  stop. 

Figure  2  portrays  cumulative  slider  data  from  the  initial  120  deg/sec,  40-second  duration 
rotation.  The  separate  lines  correspond  to  the  slider  data  for  the  separate  replications  of  all 
subjects  who  generated  usable  slider  data.  The  figure  reveals  the  general  pattern  of  subjective 
response  in  returning  the  slider  toward  neutral,  along  with  the  degree  of  variability  between 
subjects  of  that  time  course.  The  model  prediction  is  depicted  by  the  heavy  line.  Returning  to 
Figure  1,  across  subjects  and  trials,  both  the  foot  switch  and  verbal  signal  appeared  at  roughly  the 
same  time  as  the  slider  stop  position  (e.g.,  Stop  1  from  the  TSS  in  Figure  1  is  a  typical  example). 
These  two  discrete  signals  tended  to  be  more  consistent  with  each  other  than  with  the  slider 
return  to  neutral  position,  which  was  not  unexpected.  The  two  former  signals  were  therefore 
averaged  in  order  to  estimate,  for  each  subject,  on  each  run,  the  time  point  of  subjective 
stationarity  (TSS),  and  therefore  the  duration  of  washout.  Similar  procedures  were  used  to 
estimate  the  duration  of  illusory  motion  (DIM)  during  the  stationary  phase.  The  two  replications 
for  each  profile  were  averaged  to  provide  a  single  estimate  for  each  subject,  which  was  the  input 
for  statistical  analysis. 


100 


Time  (seconds) 

Figure  2.  Slider  movement  for  the  Left  120  deg/sec,  40-second  spin.  Data  are  shown  for  the  16  subjects  who 
had  usable  slider  data  for  this  particular  trial.  The  heavy  line  depicts  the  model  [11]  prediction. 


Three  separate  3x3  repeated  measures  analyses  of  variance  (ANOVAs)  were  carried  out 
on  the  three  dependent  variables  of  greatest  interest:  TSS,  DIM,  and  the  verbal  estimate  of 
subjective  magnitude  of  the  illusory  rotation.  In  each  ANOVA  the  two  factors  were  the  three 
levels  of  spin  duration  and  the  three  levels  of  plateau  spin  velocity.  All  ANOVAs  accounted  for 
missing  data  (from  some  subjects  on  some  trials),  as  the  repeated-measures  analysis  (run  using 
SPSS)  is  useful  for  both  balanced  and  unbalanced  models.  All  ANOVAs  were  accompanied  by 
Huynh-Feldt  adjustments  for  violations  of  sphericity  (when  deemed  appropriate  by  Mauchly’s 
test  of  sphericity),  and  were  corrected  where  needed. 

We  note  first,  that  in  some  cases,  some  subjects  never  reported  that  subjective  stationarity 
had  been  reached,  as  described  above.  Hence  the  number  of  observations  constituting  each  of 
the  9  data  points  varied.  Across  the  120  deg/sec  conditions,  this  number  is  34,  37  and  38,  for  10 
sec,  20  sec  and  40  sec,  respectively.  For  the  70  deg/sec  conditions  it  is  41,  34  (this  condition  was 
the  one  sampled  for  only  half  (14)  of  the  subjects.  Seventeen  stops  were  reported  for  these;  thus, 
we  doubled  the  number  to  34  to  estimate  the  number  of  stop  reports  we  expected  had  all  subjects 
experienced  this  condition),  and  42.  For  the  20  deg/sec  condition,  N  is  21,  28,  and  40.  Thus  we 
notice  in  several  cases,  particularly  for  the  shorter  and  slower  spins,  a  stop  signal  was  not  given 
because  subjective  stationarity  was  not  perceived  before  the  counter-rotation  started. 

Figure  3  shows  the  TSS  for  the  true  rotation  (signaling  the  time  of  complete  washout),  for 
those  subjects  who  gave  a  stop  signal,  as  a  function  of  the  three  levels  of  spin  rate  and  the  three 
plateau  durations  of  rotation.  An  ANOVA  reveals  the  generally  monotonic  and  additive  effect  of 
both  spin  rate  (F  [2,  8]  =  6.9,  p  <  0.02)  and  duration  (adjusted  for  sphericity  violation,  F  [1.102, 
4.41]  =  1 1.5,  p  <  0  .05)  on  TSS.  The  interaction  between  these  factors  was  not  significant  (F  [4, 
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16]  =  2.48,  p  =  0.1 1).  The  effect  of  spin  duration  is  essentially  linear  as  reflected  by  the  product 
moment  correlation  (Yaw  20,  r  =  0.97;  Yaw  70,  r  =  0.99;  Yaw  120,  r  =  0.99)  with  significant 
contrasts  between  10  and  20  seconds  (p  <  0.01)  and  between  20  and  40  seconds  (p  <  0.02). 
However,  the  effect  of  spin  rate  (yaw  velocity)  is  distinctly  non-linear,  with  a  much  smaller 
effect  (p  =  0.10)  between  the  two  slower  speeds  than  between  the  middle  and  fastest  (p  <  0.05). 


Stopl:  Time  until  Subjective  Stationarity 
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Figure  3.  Duration  of  time  until  subjective  stationarity  (TSS).  Note  that  in  some  cases,  some  subjects  never 
reported  that  subjective  stationarity  had  been  reached.  Hence  the  number  of  observations  constituting  each 
of  the  9  data  points  varied.  Across  the  Yaw  120  line,  this  number  is  34,  37  and  38,  for  10, 20,  and  40  seconds 
respectively.  For  the  Yaw  70  line,  it  is  41, 34  and  42.  For  the  Yaw  20  line,  it  is  21, 28  and  40.  Error  bars 
represent  one  standard  error  above  and  below  the  mean. 


As  was  shown  in  Figures  1  and  2,  the  form  of  the  decay  toward  this  point  of  subjective 
stationarity  is  slightly  ojival,  indicating  a  gradual  increase  in  deceleration  perception,  followed 
eventually  by  a  decay  toward  0.  We  estimated  the  decay  constant  using  only  the  40  second  spins 
because  only  here  could  we  assume  that  the  point  of  subjective  stationarity  was  consistently 
reached  before  the  deceleration  phase  of  the  profile  would  contribute  a  counter-rotation  illusion 
to  the  perceived  magnitude.  The  estimate  of  the  decay  constant  from  these  data  was  t  =  8.3  sec. 

Figure  4  depicts  the  perceived  duration  of  illusory  motion  (DIM),  following  the  true 
stationarity  of  the  simulator.  In  contrast  to  the  data  for  TSS,  nearly  all  subjects  reported  the 
second  stop,  allowing  measurement  of  DIM.  Again,  the  data  reveal  mono  tonic  effects  of  both 
spin  rate  (adjusted  F  [1.326,  10.606]  =  16.78,  p  <  0.01)  and  spin  duration  (F  [2,  16]  =  12.80,  p< 
0.01)  with  a  non-significant  interaction  effect  (F  [4,  32]  =  2.48,  p  >  0.05).  Comparisons  between 
adjacent  pairs  of  each  dependent  variable  did  not  reach  conventional  levels  of  statistical 
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significance  (p  <  0.05).  Noteworthy  is  the  fact  that,  while  the  mean  duration  of  TSS  (Figure  3) 
and  illusory  rotation  (Figure  4)  are  roughly  equivalent,  the  effect  of  increasing  spin  duration 
(calculated  as  the  difference  between  the  fastest  and  slowest  spin)  was  considerably  greater  in  the 
original  washout  (Figure  3:  10.5  sec)  compared  to  its  effect  on  the  later  illusory  rotation  (Figure 
4:  5.5  sec).  In  contrast,  the  effect  of  increasing  spin  rate  was  close  to  equivalent  across  the  two 
measures  (original  washout:  5.7  sec;  illusory  washout:  4.3  sec).  In  order  to  confirm  these 
observations,  the  two  dependent  variables  of  time  estimation  (TSS  and  DIM)  were  collapsed  into 
a  single  ANOVA  with  phase  (spin  versus  stationary)  as  an  added  factor.  This  ANOVA  revealed 
no  effect  of  phase  (the  duration  of  illusory  rotation  was  as  long  as  that  of  true  washout),  an 
interaction  between  phase  and  spin  duration  (F  [2,  8]  =  13.249,  p  <  0.05),  but  no  significant 
interaction  between  phase  and  spin  rate  (F  [2,  8]  =  4.383,  p  =  0.052).  Thus  the  effect  of  spin 
duration  is  diminished  from  true  washout  to  illusory  washout,  while  the  effect  of  spin  rate  is  not. 


Stop2:  Duration  of  Illusory  Motion 
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Figure  4.  Duration  of  illusory  motion  (DIM). 

Error  bars  represent  one  standard  error  above  and  below  the  mean. 


An  ANOVA  was  conducted  on  the  subjective  magnitude  of  the  illusory  rotation  whose 
data  are  shown  in  Figure  5.  The  ANOVA  revealed  no  effect  of  duration  (F  [2,  18]  =  0.48),  nor  of 
spin  rate  (F  [2,  18]  =  0.60),  but  a  significant  effect  of  spin  rate  and  duration  interaction  (F  [4,  36] 
=  2.69,  p  <  0.05).  This  interaction  reveals  that  the  increasing  duration  of  the  initial  spin 
increased  illusory  magnitude  for  the  two  fastest  spin  rates  (and  more  so  for  the  fastest  spin  rate, 
an  expected  effect),  but  had  no  effect  on  illusion  magnitude  for  the  slowest  spin.  In  considering 
why,  at  10  sec  duration  the  Yaw  20  spin  produced  larger  illusory  ratings  than  the  faster  spins,  it 
is  important  to  recall  that  these  ratings  were  given  relative  to  relative  to  the  initial  (actual)  spin, 
rather  than  on  an  absolute  scale.  Importantly,  the  overall  magnitude  of  experienced  illusory 
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rotation  is  roughly  half  that  of  the  initial  rotation  (M  =  5.8  on  a  scale  of  1  to  10,  where  10  was 
assigned  to  estimate  the  rate  of  the  initial  (actual)  spin),  and  this  attenuation  factor  remains  little 
effected  by  the  original  rotation  rate. 


Figure  5.  Magnitude  of  illusory  rotation. 

Error  bars  represent  one  standard  error  above  and  below  the  mean. 


Finally,  analyses  were  conducted  on  the  three  dependent  variables,  now  dividing  the 
subjects  into  pilots  and  non-pilots,  thereby  creating  a  third  “experience”  variable,  to  be  coupled 
with  plateau  duration  and  spin  magnitude.  Importantly,  there  was  neither  a  main  effect  of 
experience,  nor  any  significant  interactions  between  experience  and  profile  parameters  (all  p’s  > 
0.10),  suggesting  that  both  groups  were  influenced  equivalently  by  the  spin  dynamics. 

DISCUSSION 

The  fundamental  assumption  underlying  this  research  is  that  two  primary  parameters  will 
affect  the  perceived  rotations  in  our  subjects:  the  true  rotation  of  the  aircraft,  and  the  time 
allowed  to  reach  a  constant  rate  of  rotation.  This  speed  of  semicircular  canal  fluid  rotation, 
whether  it  reaches  washout  (fluid  rotation  equal  to  skull  rotation)  or  not,  will  determine  both  the 
magnitude  and  duration  of  illusory  counter-rotation,  once  deceleration  has  begun  and  true 
rotation  has  ceased.  In  the  current  data,  it  is  apparent  that  neither  the  10-  nor  20-second  spin 
duration  allowed  asymptote  (washout)  to  be  reached.  Had  it  been,  then  the  40-second  duration 
would  have  produced  no  further  increase  in  illusory  magnitude,  relative  to  the  20-second 
duration.  So  both  independent  variables  that  contributed  to  the  faster  semicircular  canal  fluid 
rotation  rate  (as  assessed  by  the  longer  washout  estimate)  led  to  a  consistently  longer  counter- 
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rotation  illusion.  However  the  spin  duration  effect  on  time  to  subjective  stationarity  (TSS) 
declined  more  from  the  original  rotation  (Figure  3)  to  the  illusory  rotation  (Figure  4)  than  did  the 
spin  rate  effect.  The  linear  correlation  between  the  two  duration  measures  (TSS  and  DIM)  was 
indeed  quite  high  (r  =  0.85). 

Given  the  pattern  of  data  shown  in  Figures  3,  4  and  5,  it  is  appropriate  to  discuss  some  of 
the  departures  from  a  linear  additive  model  of  spin  duration  and  magnitude  that  were  observed. 
First,  we  note  the  interaction  in  Figure  5,  whereby  with  low  spin  rates  (the  dotted  line), 
increasing  spin  plateau  duration  had  no  effect  on  illusory  magnitude  (actually  a  non-significant 
decrease),  but  the  plateau  duration  effect  emerged  and  was  progressively  greater  as  rates 
increased  to  70  and  then  to  120  deg/sec.  This  effect  was  predicted,  because  we  assume  that  the 
decay  or  washout  from  the  slow  spin  (dotted  line)  was  already  essentially  completed  relatively 
rapidly,  so  that  increasing  duration  from  10  to  20  to  40  seconds  did  nothing  to  increase  the  sense 
of  counter-rotation.  With  greater  spin  rate  magnitudes  of  70  and  120  deg/sec,  however,  decay 
would  be  less  than  fully  complete  after  20  seconds,  and  particularly  after  10  seconds,  so  that  the 
longer  spin  durations  would  allow  washout  to  occur  at  a  faster  angular  velocity  (and  thereby 
increase  the  magnitude  of  illusory  counter-  rotation).  It  remains  somewhat  puzzling  why  this 
pattern  of  effects  (actually  the  absence  of  a  duration  effect  for  the  slowest  spin)  was  manifest 
only  in  the  subjective  magnitude  of  the  illusion  and  not  the  duration.  The  most  noteworthy 
departure  from  linearity  is  the  nonlinear  effect  of  spin  rate  on  the  initial  washout  time  (Figure  3 
TSS),  such  that  only  small  differences  are  observed  between  the  two  slower  rates,  but  a  large 
increase  is  observed  from  the  middle  to  the  fastest  rate. 

The  current  data  also  served  to  address  another  important  goal  of  the  experiment, 
revealing  a  fairly  consistent  estimate  of  the  washout  decay  time  constant  (t  =  8.3  sec)  across  spin 
rates  for  the  longest  (40  sec)  plateau,  within  which  such  a  decay  could  be  readily  measured  in 
most  subjects,  as  the  point  of  subjective  stationarity  (complete  washout)  had  been  reached.  This 
value  suggests  a  faster  rate  than  that  estimated  on  the  basis  of  nystagmus  responses  [5,  Figure 
21].  This  parameter  has  been  incorporated  into  our  model  [11],  as  we  now  discuss. 

In  order  to  evaluate  the  conformance  of  the  current  experimental  data  set  with  predictions 
from  the  computational  model  of  perceived  yaw  axis  velocity  [11]  which  integrates  the  two 
illusory  phenomena  (washout  and  counter-rotation),  parameters  of  the  nine  conditions  coupled 
with  the  best  fit  8.3  sec  decay  constant,  were  input  to  the  model.  The  model  then  generated 
curves  such  as  that  shown  in  Figure  2,  representing  the  model-predicted  “slider  data”  or 
subjective  perception  of  rotation.  From  these  curves  were  extracted  the  computational  measure 
of  illusory  magnitude.  We  wished  to  correlate  these  with  the  subject-generated  ratings  of 
illusory  magnitude.  However  the  values  depicted  in  Figure  5  are  those  of  relative  magnitude, 
always  nonnalized  against  the  value  of  10  that  was  assigned  as  the  standard  for  the  original 
rotation,  whether  this  was  20,  70,  or  120  deg/sec.  Had  we  instead  requested  absolute  ratings, 
there  is  no  doubt  that  the  three  rotation  speed  triads  would  have  been  offset  accordingly,  with  the 
“Yaw  20”  (20  deg/sec)  line  in  Figure  5  much  lower,  and  the  “Yaw  120”  line  much  higher.  We 
estimated  the  amount  of  these  offsets  on  the  basis  of  Stevens’  law  of  psychophysical  rotation 
magnitude  estimation,  in  which  an  exponent  of  1.3  relates  perceived  to  actual  magnitude  [12]. 
Applying  the  correction,  in  order  to  create  an  estimated  perceived  absolute  magnitude  (lowering 
and  raising  the  two  lines  in  the  graph  by  proportionality  given  by  the  1 .3  exponent),  we 
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correlated  these  estimated  perceived  rotation  rates,  with  the  model  predictions,  and  observed  a 
correlation  of  r  =  +0.966,  a  strong  validation  of  the  success  of  the  model  predictions. 


We  did  not  attempt  to  use  the  model  to  predict  the  two  time  measures,  for  the  reason  that 
such  prediction  would  have  required  estimating  a  given  threshold  at  which  the  model  would  have 
reported  a  “stop”.  Because  such  a  threshold  occurs  at  a  very  low  level  of  perceived  rotation  and 
therefore  at  a  point  at  which  the  decay  curve  is  nearly  flat,  a  very  slight  change  in  this  threshold 
(e.g.,  4%  to  3%  of  maximum  assumed  to  be  the  threshold)  would  cause  very  large  changes  in  the 
model-predicted  stop  time.  Since  we  had  no  objective  way  of  assessing  what  threshold  value 
subjects  actually  used  to  say  “stop”,  we  had  little  confidence  in  basing  this  model  on  any 
particular  value. 


Our  final  goal  was  to  establish  the  extent  to  which  some  level  of  flight  experience  might 
mitigate  or  alter  the  strength  of  the  two  phenomena  (washout  and  illusory  counter-rotation). 
Clearly  the  current  data  do  not  support  this  hypothesis.  Despite  the  relatively  high  statistical 
power  of  the  experiment  and  the  consistently  observed  statistical  effects  of  other  variables,  the 
analysis  by  experience  failed  to  reveal  any  effects  that  were  even  “close”  to  statistical 
significance  (i.e.,  using  the  more  liberal  criterion  of  0.10).  Such  a  finding  (although  tempered  by 
the  fact  that  we  did  not  include  any  high-time  fighter  pilots  in  our  sample)  is  certainly  consistent 
with  the  evidence  from  mishaps  and  spatial  disorientation  incident  reports,  that  spatial 
disorientation  problems  do  not  disappear  with  experienced  pilots  [8]. 

In  conclusion,  the  current  research  sought  to  understand  and  quantify  two  of  the  illusory 
effects  that  are  often  involved  in  a  pilot’s  misperception  of  aircraft  state,  leading  to  the  leans  and 
eventually  to  the  grave  yard  spiral.  These  are  the  “washout”  associated  with  a  sustained  rotation, 
and  the  illusory  counter-rotation  when  that  true  rotation  has  ceased.  To  this  can  be  added  a  third 
illusory  component,  not  examined  here,  but  critical  in  initiating  the  first  two:  that  is,  the  failure 
of  pilots  to  notice  sub-threshold  rolls  (rotation  around  the  fuselage  axis  of  the  airplane),  that  can 
initiate  the  rotation  around  the  vertical  axis.  Our  data  were  the  first  to  describe  both  of  the  first 
two  effects,  acting  in  consort,  in  a  way  that  would  mimic  the  pilot  intentionally  trying  to  come 
out  of  a  sustained  tight  turn,  and  we  observed  that  all  9  conditions  generated  substantial  illusions 
of  both  sorts.  Importantly,  we  were  also  able  to  predict  much  of  the  variance  in  illusory 
experience  from  a  fairly  simple  computational  model. 
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Abstract 

In  a  low-fidelity  fixed-base  F-16  flight  simulator,  12  fighter  pilots  attempted  to  recover  from 
unusual  pitch-down  inverted  attitudes.  Recovery  was  done  in  a  control  condition,  aided  by  a 
HUD  only,  and  with  three  command  display  augmentations:  (1)  a  command  visual  icon, 
pointing  in  the  direction  of  appropriate  control;  (2)  the  icon  augmented  with  a  voice  command; 
and,  (3)  augmented  with  a  tactile  command.  All  three  command  displays  reduced  the  time  to 
make  the  initial  recovery  response  relative  to  the  control  condition,  and  decreased  the  frequency 
of  initial  incorrect  roll  responses.  The  tactile  and  voice  augmentations  also  improved  the  speed 
of  initial  recovery  response  from  the  most  severe  inversions.  Although  the  tactile  command 
effectively  supported  performance,  it  was  the  least  preferred  of  the  three  display  augmentations. 


Introduction 

Spatial  disorientation  (SD)  continues  to  be  an  enduring  problem  in  aviation.  For  example,  during 
the  1980’s  and  90’s,  SD  was  a  factor  in  approximately  21%  of  all  US  Air  Force  Class  A* 
mishaps.  Between  October  1993  and  September  2002,  there  were  25  high  perfonnance  fighter  or 
attack  mishaps  where  SD  was  identified  as  a  causal  or  contributing  factor.  These  mishaps 
resulted  in  19  fatalities  and  cost  the  Air  Force  over  $455  million  (Sundstrom,  2004).  In  the 
United  Kingdom,  SD  mishap  frequency  is  estimated  to  be  between  6  and  21%  (depending  on 
branch  of  service)  (Holmes  et  ah,  2003).  Furthennore,  SD  accidents  are  often  fatal.  In  civilian 
aviation,  Wiegmann  et  al.  (2004)  observed  that  over  90%  of  SD  accidents  resulted  in  fatalities. 

In  the  current  research  we  are  most  directly  interested  in  the  loss  of  attitude  awareness  (knowing 
which  way  is  up,  and  the  orientation  of  key  spatial  landmarks). 

Not  surprisingly,  along  with  training  (Braithwaite,  2004),  research  has  focused  on  display 
changes  and  augmentations  that  may  prevent  SD  accidents  either  by  aiding  recovery  from  SD,  or 
by  preventing  SD  from  occurring  in  the  first  place.  Some  display  approaches  focus  directly  on 
visual  displays.  For  example  debates  on  the  appropriate  frame  of  reference  for  attitude  indicators 
are  based,  in  part,  on  the  fact  that  the  inside-out  moving  horizon  attitude  indicator  more  readily 
invites  inappropriate  corrections  when  in  unusual  attitudes  (Roscoe,  2004;  Kovalenko,  1999; 
Previc  and  Ercoline,  2001).  Recent  advances  in  visual  display  augmentations  have  also  focused 
on  incorporation  of  augmenting  visual  cues  (typically  on  a  HUD)  to  indicate  the  direction  of  the 


*  A  Class  A  mishap  is  defined  as  one  where  there  is  loss  of  life,  injury  resulting  in  permanent  total  disability, 
destruction  of  an  AF  aircraft,  and/or  property  damage/loss  exceeding  $1  million  (Air  Force,  2001). 
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horizon  (Hawarth  and  Evans,  2004;  Reising,  Liggett  and  Munns,  1999),  or  to  command  the 
direction  of  stick  movement  that  will  restore  level  flight  from  an  unusual  attitude  (Wickens  et  al, 
in  press). 

An  alternative  approach  is  through  sensory  augmentation,  whereby  added  auditory  or  tactile 
information  is  presented  to  preserve,  or  restore,  orientation.  Such  augmented  cues  have  the 
advantage  of  supporting  perceptual  processing  even  if  the  visual  channel  is  overloaded,  is 
tunneled  on  a  particular  indicator  (e.g.,  the  attitude  indicator),  or  is  degraded  because  of  high  g 
forces  (e.g.,  Erp,  Veltman,  &  van  Veen,  2003,  Eriksson  et  al,  2006). 

In  considering  either  auditory  or  tactile  cues,  a  critically  important  distinction  is  between  status 
displays,  that  infonn  the  pilot  where  the  horizon  (or  some  other  orientation  landmark)  currently 
is,  and  command  displays,  that  inform  the  pilot  of  an  action  to  take  to  restore  an  appropriate 
orientation.  There  are  numerous  differences  in  these  two  display  philosophies  (Andre  and 
Wickens,  1992;  Wickens  and  Hollands,  2000),  many  of  which  are  somewhat  peripheral  to  the 
current  issue.  However  one  key  difference  is  that,  in  time-critical  situations,  so  long  as  the 
reliability  of  the  automation-induced  action  command  is  high,  command  displays  will  be  more 
effective  than  status  displays  (Sarter  and  Schroeder,  2001).  This  is,  in  large  part,  because  the 
command  display  eliminates  a  time-consuming  and  potentially  error-prone  human  decision 
regarding  what  to  do  when  an  inappropriate  status  (e.g.,  attitude)  exists. 

The  distinction  between  status  and  command  displays  is  also  relevant  to  the  distinction  between 
displays  designed  to  prevent  SD  in  the  first  place  (status  displays  are  typically  used;  e.g.,  that 
shows  where  the  horizon  actually  is),  and  those  designed  to  recover  from  existing  SD.  In  the 
latter  case,  given  the  time  criticality  of  the  adverse  state,  command  displays  are  preferable,  and 
hence  such  command  displays  represent  the  focus  of  the  current  experiment,  comparing  visual 
with  auditory  and  with  tactile  command  displays. 

As  noted,  a  visual  command  display  can  often  be  represented  as  an  arrow  or  pointer  commanding 
an  action.  With  an  auditory  display,  the  most  compatible  way  of  unambiguously  representing  an 
action  is  through  voice  command  (e.g.,  “pull  up”),  rather  than  through  auditory  spatial  cues  (e.g., 
Begault  and  Wenzel,  1992,  Brickman  et  al,  2000).  Voice  can  directly  articulate  the  verb  which 
signals  action,  whereas  a  tone  cannot,  as  it  is  more  ambiguous.  Does  a  tone  in  the  left  ear 
indicate  that  one  should  pull  to  the  left,  or  that  danger  is  on  the  left  (hence  pull  to  the  right)?  In 
fact,  it  may  be  argued  that  a  verbal  auditory  command  is  even  less  ambiguous  than  its  visual 
counterpart,  the  arrow  or  pointing  icon.  Indeed,  the  authors  have  recordings  of  two  SD  incidents 
where  a  person  outside  the  incident  aircraft  helps  the  incident  pilot  recover  via  instructive  radio 
calls. 

In  contrast  to  voice,  tactile  displays  are  sufficiently  new  in  their  airspace  implementation  that 
there  are  no  firm  guidelines  with  regard  to  presenting  status  or  command  information.  Indeed 
there  appears  to  be  a  mixture  of  display  types  proposed  (Van  Erp  et  al.,  2006;  Sarter,  in  press). 
For  example  some  of  the  best  evaluated  systems  appear  to  use  tactile  status  displays  (e.g., 

Rupert,  2000),  indicating  for  example  a  constant  heading,  orientation,  or  gravitational  vector, 
location  and  severity  of  wing  icing  (McGuirl  and  Sarter,  2001),  or  automation  changes  (Sklar 
and  Sarter,  1999).  With  few  exceptions,  these  have  not  been  compared  directly  with  other  non- 
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visual  modality  displays.  A  study  by  Calhoun  et  al  (1993)  compared  tactile  with  auditory 
alerting,  as  redundant  channels  with  a  visual  alert,  and  revealed  that  both  non-visual  modalities 
improved  performance  equally.  A  study  by  Ho  &  Sarter  (2004)  found  qualitative  differences 
between  the  three  (auditory,  visual,  tactile)  channels  as  communications  media.  More  recently, 
van  Erp  et  al.  (2006)  have  directly  compared  command  versus  status  tactile  displays  for  recovery 
of  constant  heading,  when  spatial  orientation  is  lost  due  to  the  effects  of  the  somatogyral  illusion 
(e.g.,  the  feeling  of  turning  following  cessation  of  a  constant  prolonged  turn).  In  a  between- 
subjects  design,  they  found  no  difference  in  effectiveness  between  these  two  forms  of  sensory 
augmentation,  although  they  found  that  both  were  superior  to  a  control  condition  in  terms  of  their 
ability  to  restore  straight  flight. 

Actual  comparisons  between  auditory  and  tactile  command  information  as  deployed  in  a 
command  mode  for  unusual  attitude  recovery  do  not  appear  to  have  been  conducted,  and  such  a 
comparison  is  the  objective  of  the  current  study.  Here  pilots  flying  a  low-fidelity  fixed-base  F- 
16  simulator  implemented  with  Microsoft  FlightSim  are  diverted  into  an  unusual  attitude  and 
then,  depending  on  conditions,  requested  to  recover  as  rapidly  as  possible,  either  (a)  unaided 
(using  only  the  status-based  HUD  [although  augmented  as  in  the  real  airplane  with  a  pull-up 
voice  command  when  within  a  short  time  of  ground  impact]),  (b)  aided  by  a  visual  icon  alone 
(pointer  in  the  direction  of  the  necessary  control  movement(s),  (c)  aided  by  voice  commands  + 
visual  icon  and  (d)  aided  by  tactile  “action”  commands  +  visual  icon,  whereby  the  movement  of 
apparent  sensation  across  the  torso  signaled  the  direction  of  required  roll  (lateral  movement)  or 
pitch  correction  (vertical  movement). 


Method 


Equipment 

The  equipment  consisted  of  a  Dell  3.2  GHz,  Pentium  4  processor  with  a  30”  liquid  crystal 
display  (LCD)  monitor.  In  addition,  a  Thrustmaster  Cougar  stick  and  throttle  was  used  to 
simulate  the  F-16  input  devices.  Software  included  Microsoft  FlightSim  2004  with  an  approved 
F-16  model  downloaded  from  the  Microsoft  Flight  Simulator  Developer  web  site  (found  at: 
http://www.lagosim.com/?act=scheda&p=100034&lang=eng&undersearch=fs%20falcon).  The 
vibro-tactile  vest  was  made  by  TNO,  a  company  in  Holland.  The  vest  contains  64  tactors  that 
could  be  programmed  to  give  specific  low-frequency  vibration  feedback.  Voice  commands  were 
produced  through  the  computer’s  sound  card  and  the  two  front  speakers  of  a  six-speaker  system. 
The  display/control  interface  was  positioned  within  a  fighter-aircraft  cockpit  mockup,  shown  in 
Figure  1. 

Participants 

The  participants  were  12  Air  Force  pilots  all  with  fighter  aircraft  experience.  All  12  pilots  also 
had  over  100  hours  of  HUD  experience. 

Task 

The  pilots’  task  was  to  fly  a  series  of  legs  at  a  fixed  altitude  above  ground,  and  respond  to  the 
imposition  of  unusual  attitudes  by  restoring  the  aircraft  to  a  wings  level  (within  10  degrees)  and 
pitch  level  (within  5  degrees)  attitude  at  or  above  1000  feet  radar  altitude  and  350  knots  (within 
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10  knots)  as  rapidly  as  possible.  When  the  pilot  reached  these  conditions  for  2.5  seconds,  the 
trial  terminated. 


Figure  1.  Experimental  configuration. 


To  initiate  the  unusual  attitude  (UA),  straight  and  level  flight  was  interrupted  as  the  HUD  screen 
was  blanked  for  a  period  of  2  seconds.  This  interval  of  time  was  meant  to  simulate  a  time  in 
which:  (a)  a  simulated  autopilot  flew  an  emergency  missile  evasion  profile  and  (b)  the  pilots 
eyes  were  diverted  from  the  instrument  panel.  At  the  end  of  this  interval  the  HUD  screen  was 
restored,  and  the  pilot  attempted  to  recover,  first  returning  roll  to  an  upright  attitude,  and  then 
returning  pitch  to  a  level  attitude.  Only  pitch  downward  UAs  were  imposed.  On  the  rare 
occasions  when  radar  altimeter  reached  zero  (i.e.,  the  plane  crashed),  the  trial  terminated.  We 
accepted  this  risk  because  we  wanted  the  UAs  to  be  challenging,  thus  further  inspiring  the  pilots 
to  recover  as  quickly  as  possible. 

Simultaneously  with  HUD  restoration,  three  sorts  of  command  inputs  could  be  delivered: 

(1)  As  shown  in  Figure  2,  a  quadrahedral  icon  would  appear  on  the  HUD  display  (Wickens  et 
al,  in  press)  with  extensions  that  pointed  in  the  direction  of  the  required  stick  movement 
(i.e.,  left  ->  roll  left;  up  pull  up). 

(2)  A  voice  command  would  utter  two  phrases,  a  roll  correction  (“roll  left”  or  “roll  right”)  and 
a  pitch  correction  (“pull  up”).  The  roll  command  was  repeated  with  1.11  seconds  between 
the  commands  until  an  upright  attitude  (<  90  degrees  roll)  had  been  restored,  and  then  the 
pitch  command  was  alternated  with  a  roll  command  at  the  same  rate. 

(3)  A  tactile  command.  This  was  created  by  sweeping  a  set  of  7  tactors  across  the  torso,  left-> 
right  or  rights  left,  mimicking  roll  right  and  roll  left  commands,  respectively.  The 
duration  of  the  sweep  was  0.78  seconds.  As  with  the  voice  commands,  following 
restoration  of  an  upright  roll  (<  90  degrees),  the  pitch-up  commands  were  presented  by 
sweeping  a  set  of  5  tactors  up  the  center  of  the  torso,  with  a  duration  of  0.56  seconds. 
Tactile  stimuli  began  with  a  stimulus  on  the  side  opposite  the  direction  of  desired  response, 
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and  intensified  by  means  of  more  tactile  activation  as  the  sweeping  motion  reached  the 
desired  direction  of  response.  The  icon  was  always  present  redundantly  whenever  the 
voice  or  tactile  commands  were  used;  and  the  latter  two  commands  initiated  simultaneously 
with  the  appearance  of  the  icon,  which  was  at  UA  initiation.  The  timing  of  the  tactile  and 
voice  commands  was  identical. 


Figure  2.  SD  icon  snapshots  for  an  aircraft  whose  state  is:  (a)  0°  Pitch,  0°  Roll;  (b)  -90°  (down)  Pitch,  0° 
Roll;  (c)  0°  Pitch,  -45°  (left)  Roll;  and,  (d)  -45°  Pitch,  +45°  Roll.  The  icon  in  the  figure  points  in  the 
direction(s)  to  correct  for  these  states  back  to  straight  and  level  flight. 

Each  trial  was  flown  until  a  criterion  of  straight  and  level  flight,  as  described  above,  and  then  the 
trial  tenninated.  The  pilot  was  given  approximately  1  minute  before  commencing  the  next  trial. 

Design 

An  entirely  within  subjects  design  was  used,  varying  three  factors  orthogonally  in  a  4x2x2 
design; 

1.  Four  levels  of  display  type  were  blocked  within  different  trial  sets  (control,  icon  only, 
tactile  with  icon,  voice  with  icon). 

2.  A  random  set  of  pitch  and  inverted  roll  deviations  were  assigned  to  two  fundamental 
categories:  severe  inversion  (at  about  15  degrees  from  a  complete  180-degree  roll 
inversion)  and  moderate  inversion  (at  about  60  degrees  from  complete  inversion). 

3.  Each  display  condition,  and  its  set  of  attitudes  was  encountered  in  two  blocks,  providing 
a  replication  (or  learning)  variable  of  two  levels. 

Within  each  replication  x  display  block,  four  unusual  attitudes  were  imposed.  These  included 
between  1  and  3  cases  of  severe  inversions  (and  correspondingly  between  3  and  1  cases  of 
moderate  inversion.  Typically,  there  were  2  of  each,  but  randomization  sometimes  yielded  a  3-1 
mix,  never  a  4-0  mix.  Each  of  these  four  was  also  randomly  assigned  to  4  different  levels  of 
nose-down  pitch:  30,  45,  60,  and  75  degrees. 

Practice  trials  always  contained:  [pitch  30,  roll  1 10],  [pitch  30,  roll  260],  [pitch  85,  roll  140], 
and  [pitch  85,  roll  200].  (Where  the  roll  convention  used  by  Microsoft  FlightSim  is:  upright 
level  is  0;  90  degrees  right  roll  is  90;  inverted  is  180;  left  90  is  270;  and,  left  1  degree  is  359. 
FlightSim  also  uses  a  pitch  convention  of  positive  as  nose  down.) 

Each  unusual  attitude  was  encountered  at  a  random  duration  between  10  and  30  seconds  after  the 
flight  leg  initiated.  Each  pitch  and  roll  angle  for  the  UA  was  further  randomized  by  adding  or 
subtracting  up  to  5  degrees  so  that  pilots  could  not  develop  expectations  for  an  upcoming  UA’s 
specific  angles.  For  the  first  replication,  the  four  display  conditions  were  presented  in  a  counter¬ 
balanced  order  across  all  pilots.  Then  each  pilot  encountered  the  second  four  replications  in 
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pseudo-random  Latin  square  design  (Table  1).  The  experimental  protocol  was  approved  by  the 
local  (Air  Force  Academy)  IRB. 


Table  1.  Counterbalancing  order. 
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Procedure 

Upon  arriving  at  the  experimental  facility,  pilots  read  and  signed  the  consent  form,  and  then  were 
introduced  to  the  general  procedures  of  the  experiment.  Each  pilot  was  given  10  minutes  (or  as 
long  as  they  felt  appropriate)  to  familiarize  himself  with  the  flight  dynamics,  and  then  practice 
UA  recovery  in  the  control  condition  for  4  trials,  or  as  long  as  he  felt  appropriate.  Then,  prior  to 
each  block  of  the  first  replication,  pilots  were  given  four  trials  or  more  (greater  than  four  trials 
only  happening  in  one  case)  to  practice  recoveries  with  a  particular  display  type. 

During  the  second  replication,  pilots  were  alerted  to  which  display  type  they  would  receive  prior 
to  each  block,  but  were  given  no  further  practice  trials.  Following  the  experiment,  pilots  were 
asked  to  complete  a  brief  form  requesting  subjective  ratings  of  each  display,  and  inviting  other 
comments.  The  entire  experiment  required  approximately  90  minutes  to  complete. 

Data  Analysis 

Data  were  analyzed  in  terms  of  a  series  of  mission  completion  and  flight  safety  measures, 
including  the  time  to  restore  stable  parameters  (see  above),  minimum  radar  altitude  flown, 
maximum  g  force,  minimum  time-to-impact  the  terrain,  and  maximum  negative  vertical  velocity. 
In  addition,  the  continuous  measures  of  roll  and  pitch  were  evaluated  during  the  first  10  seconds 
following  the  display  re-appearance  of  each  UA  event,  and  visual  analysis  was  used  to  determine 
the  time  and  correctness  of  initiation  of  a  roll  correction,  and  of  the  subsequent  pitch  correction. 
Across  all  of  these  eight  dependent  measures,  data  were  pooled  across  the  different  pitch  upsets, 
as  well  as  the  1-3  instances  of  severe  or  moderate  inversion  within  each  replication.  All  data 
were  examined  for  nonnality,  transformed  if  a  high  skew  value  required  it,  and  subjected  to 
SPSS  repeated  measures  ANOVAs.  Where  necessary,  subsequent  planned  contrasts  examined 
directional  hypotheses  that  all  display  augmentations  would  be  superior  to  the  control  condition, 
and  that  the  two  non-visual  augmentations  would  be  superior  to  the  icon  alone. 
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Results 

The  initial  roll  response  time  data  are  shown  in  Figure  3,  as  a  function  of  display  condition  and 
roll  inversion  severity.  Analysis  of  these  data  revealed  a  significant  main  effect  of  display  (F 
3,33  =  9.00,  p  <  0.01),  a  marginally  significant  effect  of  severity  (F  1,11  =  3.51,  p  =  0.09)  and  a 
significant  display  X  severity  interaction  (F  3,33  =  3.58,  p  =  0.02).  Planned  comparisons 
revealed  that  initial  response  times  (RTs)  were  longer  in  the  control  condition  than  in  the  three 
display  augmentation  conditions,  but  that  the  latter  three  did  not  differ  from  each  other.  There 
was  also  a  main  effect  of  training  replication  (F  1,11=  5.55,  p  <  0.05),  indicating  a  small  (less 
than  0. 1  second)  reduction  in  RT  from  the  first  to  the  second  replication. 


Mean  Roll  Response  Time 
by  Roll  Inversion  and  Condition 


Aural  Tactile  Icon  Control 

Condition 


|  ri  Moderate  Inversion  BSevere  Inversion  | 

Figure  3.  Initial  response  time  as  a  function  of  display  condition  and  severity  of  roll  inversion  (white: 

modest,  black:  severe). 

In  order  to  examine  the  significant  interaction  in  more  detail,  the  data  in  the  “severe”  inversion 
condition  (at  about  15  degrees  from  inverted)  were  submitted  to  a  two-way  2  (replication)  x  4 
(condition)  ANOVA.  As  would  be  expected,  this  ANOVA  again  revealed  a  significant  effect  of 
display  (F  3,33  =  9.30,  p  <  0.01).  However  in  contrast  to  the  ANOVA  performed  on  the  full  data 
set  (i.e.,  including  the  moderate  inversion),  this  ANOVA  revealed  a  clear  advantage  of  the  voice 
and  tactile  conditions  over  the  icon-only  condition.  A  planned  contrast  of  the  two  former  versus 
the  latter  yielded  a  significant  benefit  of  the  non-visual  augmentation  (p  =  0.04),  as  seen  in  the 
black  bars  of  Figure  3.  This  benefit  was  not  present  with  the  moderate  inversion  (i.e.,  at  about  60 
degrees  from  inverted). 

In  addition  to  initial  roll  response  time,  we  also  examined  the  frequency  of  roll  reversal  errors, 
characterized  by  an  initial  roll  rotation  in  the  opposite  direction  to  that  which  was  commanded 
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(or  was  appropriate,  in  the  case  of  the  control  condition).  Such  errors  were  committed  on  only  a 
small  proportion  of  the  trials  (14/384),  and  were  distributed  as  follows:  Voice  2,  Tactile  4,  Icon- 
only  1,  and  Control  8.  A  chi-squared  test  revealed  that  this  distribution  significantly  (p  =  0.05) 
departed  from  the  hypothesis  of  equal  proportions.  Thus,  as  with  initial  response  time,  so 
accuracy  appeared  to  benefit  by  the  presence  of  a  command  display. 

Pitch  response  time  data  were  about  2  seconds  longer  and  less  consistent  than  the  roll  response, 
reflecting  the  fact  that  pilots  appropriately  rolled  to  an  upright  attitude  before  initiating  their 
pitch  recovery.  Perhaps  in  part  because  of  these  factors,  initial  response  time  for  pitch  failed  to 
reflect  significant  effects  of  any  of  the  independent  variables,  nor  their  interactions. 

The  measure  of  overall  time  to  restore  straight  and  level  flight  also  failed  to  reflect  significant 
effects,  as  these  times  were  quite  long  (averaging  around  40  seconds),  and  clearly  reflected  the 
contribution  of  numerous  factors  in  addition  to  the  specific  variables  manipulated  in  the 
experiment. 

Of  the  four  measures  of  safety,  two  reflected  a  significant  (p  <  0.05)  effect.  First,  the  minimum 
radar  altitude  indicated  that  pilots  descended  420  feet  lower  when  recovering  from  a  severe 
rather  than  a  moderate  roll  inversion  (F  1,11=14.13,  p  <  0.01).  Further,  an  interaction  between 
training  replication  and  display  condition  (F  3,33  =  2.85,  p  =  0.05)  indicated  that,  earlier  in 
practice  (replication  1),  the  tactile  and  icon  display  allowed  maintenance  of  a  higher  minimum 
altitude  than  the  control  and  auditory  displays;  but  in  later  trials  (replication  2)  all  three 
augmentations  produced  lower  descents  than  did  the  control  condition.  Second,  the  minimum 
time-to-impact  measure  revealed  a  lower  time  with  severe  (5.8  sec)  than  moderate  (6.7  sec) 
inversions  (F  1,11  =  5.46.  p  <  0.05).  While  there  was  no  effect  of  display  condition  (F  <  1),  an 
interaction  between  condition  and  practice  (F  3,33  =2.92;  Huynh-Feldt  correction  for  sphericity, 
p  <  0.05)  revealed  that  early  in  practice,  the  tactile  and  control  condition  led  to  a  greater  time; 
whereas  later  in  practice,  the  pattern  reversed,  with  the  voice  and  icon  leading  to  a  greater 
minimum  time-to-impact.  Finally,  with  regard  to  the  ultimate  measure  of  safety,  it  is  noteworthy 
that  the  crash  rate  (minimum  radar  altitude  =  0)  was  quite  low,  with  only  9  crashes  out  of  384 
opportunities.  These  crashes  were  relatively  evenly  distributed  across  the  four  conditions 
(2, 2, 2, 3). 

Finally,  subjective  ratings  of  display  preference  were  analyzed.  A  t-test  revealed  that  the  overall 
evaluation  of  the  tactile  display  was  significantly  lower  than  the  icon  display  (t22  =  2.20,  p  < 
0.05)  and  lower  than  the  voice  display  (tl9  =  2.88,  p  <  0.05).  Of  those  who  favored  one  display 
over  another,  eight  pilots  rated  the  voice  display  as  most  preferred;  two  rated  the  icon  display  as 
most  preferred,  while  none  rated  the  tactile  display  as  most  preferred. 


Discussion 

The  current  study  set  out  to  evaluate  the  effectiveness  of  attitude  command  displays,  with  and 
without  redundant  non-visual  augmentations,  to  facilitate  recovery  from  unusual  attitudes  for 
experienced  Air  Force  fighter  pilots.  The  results  indicated  that,  while  such  displays  were 
effective,  their  effectiveness  was  confined  to  the  initial  roll  response  to  an  inverted  attitude. 

Here  the  results  indicated  that  all  three  command  displays  were  helpful  in  tenns  of  both  time  and 
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accuracy;  and  furthennore  those  with  redundant  non-visual  augmentations  (tactile  and  voice 
commands)  were  particularly  helpful  in  initiating  the  response  when  the  inverted  attitude  was 
severe  (nearing  180  degrees).  In  this  regard,  the  data  are  consistent  with  the  findings  of  Calhoun 
et  al.  (2003)  that  a  pure  alerting  (i.e.,  non-directional)  system  benefited  from  redundant  aural  or 
tactile  display,  as  well  as  those  of  Van  Erp  et  al.  (2006)  who  found  in  a  spatial  disorientation 
context,  that  a  directional  tactile  alert  improved  performance  over  a  control  condition.  Our  study 
combined  these  results  by  identifying  the  equal  support  of  both  tactile  and  voice-based 
directional  commands,  in  a  fairly  realistic  flight  scenario,  with  skilled  fighter  pilots. 

In  interpreting  these  results,  it  is  important  to  note  that  the  two  sensory  augmentations  offered 
neither  earlier  nor  additional  information  to  the  command  icon,  but  did  provide  a  performance 
advantage,  suggesting  that  the  greater  salience  of  these  two  modalities  was  somewhat  more 
effective  in  capturing  attention,  relative  to  a  visual-only  cue.  It  is  also  important  to  note  that  the 
tactile  cueing  was  everywhere  just  as  effective  as  the  voice  display,  in  spite  of  the  fact  that  voice 
commands  are  far  more  familiar  than  the  tactile  commands.  In  this  regard,  the  data  speak  to  the 
potential  value  of  the  tactile  system.  Finally,  we  note  that  the  tactile  cue  was  the  least  favored  in 
terms  of  subjective  ratings,  and  three  of  the  pilots  had  strong  negative  comments  to  offer 
regarding  its  potential  value.  The  observation  that  display  augmentations  which  improve 
performance  are  not  always  preferred  is  one  that  has  often  been  observed  in  human  factors 
(Andre  and  Wickens,  1995). 

While  the  initial  response  speed  and  accuracy  benefited  from  command  displays,  we  might  also 
have  expected  that  some  of  the  benefits  of  augmentation  (and  particular  non-visual 
augmentation)  would  have  been  reflected  in  other  variables  of  greater  duration  (e.g.,  eventual 
time  to  full  recovery,  safety  measures).  The  fact  that  they  did  not  consistently  do  so  (minimum 
altitude,  time-to-impact),  or  do  so  at  all,  indicates  that  the  relatively  small  gains  in  response  time 
(tenths  of  a  second),  were  masked  by  the  considerable  variability  imposed  by  other  factors  in  this 
time  following  initiation  of  the  first  roll. 

In  spite  of  this  elimination  of  command  display  benefits  for  the  longer  duration  measures,  one 
can  envision  many  circumstances  in  which  slight  gains  in  response  time  (and,  in  particular, 
elimination  of  initial  roll  reversal  errors)  could  impose  substantial  safety  benefits.  Furthermore, 
it  can  be  asserted  that  in  some  respects  the  current  simulation  was  designed  to  make  recovery 
relatively  easy,  compared  to  operational  circumstances.  For  example  pilots  expected  an  inverted 
UA  on  every  trial,  and,  for  experimental  reasons,  this  always  produced  a  pitch-down  attitude. 
Also,  pilots  were  only  engaged  in  a  single  task  (notice  and  recover  from  UA),  whereas  in  actual 
combat,  there  could  have  been  several  concurrent  tasks  competing  for  their  attention;  for 
example,  visual  scanning  for  a  target,  auditory  communications  with  a  wingman,  auditory 
processing  of  in-cockpit  warnings.  In  particular,  the  latter  two  auditory  inputs  might  have  placed 
the  current  voice  cuing  at  a  disadvantage,  compared  to  the  tactile  cueing.  In  short,  we  would 
predict  that  if  either  or  both  of  these  factors  (expectancy,  concurrent  task  load)  were  altered  to 
more  closely  mimic  real  world  combat  situations,  the  advantages  of  command  displays,  of  non¬ 
visual  command  displays,  and  particularly  of  tactile  command  displays,  would  be  elevated 
considerably. 
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Conclusion 

In  conclusion,  the  current  results  strongly  point  to  the  advantage  of  command  displays  to 
facilitate  recovery  from  unusual  attitudes,  and  suggest  that  research  on  voice  and  tactile  displays 
be  pursued  under  conditions  when  multi-task  workload  and  low  expectancies  are  created,  as  well 
as  those  in  a  motion  based  simulator,  imposing  vestibular  disorientation  and  high  g-load 
(Erickson  et  al,  2006).  Here  we  might  expect  these  advantages  will  amplify. 
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ISAP  07  SDO  submission  (Wickens) 

Vestibular-based  spatial  disorientation  (SD)  represents  a  recurring  and  potentially  fatal  problem 
in  all  aspects  of  aviation.  For  example,  the  US  Air  Force’s  costs  associated  with  SD  have 
averaged  over  $40  million  per  year  over  a  recent  10-year  period  (Sundstrom,  2004).  The  Army, 
Navy,  and  civilian  aviation  experience  similar  SD  costs  (Sundstrom,  2004).  While  the  problem 
is  not  as  extensive  in  airline  operations,  this  domain  is  also  not  without  examples  of  severe 
disorientation  (e.g.,  recent  Dubai  crash),  and  the  consequences  of  such  accidents  are  tragically 
high. 

In  our  proposed  presentation,  we  overview  the  collective  results  of  a  3 -year  effort,  carried  out  for 
the  Air  Force  ( Multisensory  Integration  for  Pilot  Spatial  Orientation,  AFRL  contracts  F33615- 
03-M-6360  and  FA8650-04-C-6457),  to  study  spatial  disorientation,  with  particular  focus  on 
fighter  pilots,  but  generalizing  to  broader  aviation  domains.  Our  paper  will  focus  on  four  areas: 
developing  a  computational  model  of  vestibular  orientation,  validating  this  model  through 
empirical  research,  deploying  the  model  in  an  on-line  tool  to  prevent  vestibular  disorientation, 
and  deploying  it  in  an  analytical  tool  for  mishap  investigation. 

The  Vestibular  Model.  Following  the  classic  work  of  Gillingham  and  others  (1993),  we  have 
integrated  the  existing  literature  on  the  response  of  both  the  semi-circular  canals,  and  the  otolith 
organs,  to  develop  a  computational  model  of  how  aircraft  motions  yield  vestibular-based 
sensations  of  aircraft  attitude  that  are  discrepant  from  the  actual  attitude  (Small  et  al,  2005). 
Particular  emphasis  has  been  placed  on  modeling  4  illusions:  the  leans,  the  graveyard  spiral,  the 
somatogravic  illusion,  and  the  Coriolis  illusion. 

Experimental  Validation.  An  experiment  to  validate  a  critical  component  underlying  both  the 
leans  and  somatogravic  illusion  model  was  conducted  within  a  motion-based  simulator,  in  which 
the  vestibular  washout  and  the  illusory  counter-rotation  of  the  semi-circular  canals  were 
examined.  Experimental  results  comparing  actual  motions  to  the  perceived  motions  provided 
data  for  fine  tuning  the  model  parameters. 

SPAS  Intervention  Tool.  A  validated  model  could  be  implemented  in  the  cockpit,  receiving 
inputs  from  aircraft  attitude  and  some  aspects  of  pilot  control  behavior  to  infer  the  degree  of 
spatial  disorientation  (discrepancy  between  perceived  and  actual  attitude).  Depending  on  the 
degree  of  inferred  disorientation,  graded  interventions  would  be  adaptively  implemented  to 
restore  orientation.  In  one  experiment  (Wickens  et  al.,  in  press),  we  examined  the  viability  of 
less  intrusive  HUD  located  visual  icons  for  such  restoration.  In  a  second  experiment  we 
evaluated  the  viability  of  more  intrusive  voice  and  tactile  interventions  that  are  designed  to  be 
triggered  if  the  icons  fail. 

SPAT  Analysis  Tool.  Continuous  flight  data  available  post-mishap  from  HUD  tapes,  aircraft 
data  recorders,  and  ATC  radar  data  have  been  input  to  a  Spatial  Disorientation  Analysis  Tool 
(SDAT)  that  we  developed  to  reveal  the  likelihood  that  pilots  experienced  different  states  of 
disorientation,  during  the  progression  of  the  mishap  flight.  This  tool  has  been  successfully 
applied  to  characterize  the  spatial  disorientation  of  specific  mishaps  such  as  Air  Force  F-16s,  a 
Navy  transport,  and  several  general  aviation  accidents. 
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Appendix  B.  Results  of  SDAT  Heuristic  Evaluation 


A  Heuristic  Evaluation  of  the  SDAT  user  interface  was  performed  to  evaluate  its  usability. 

Three  usability  experts  were  consulted.  Their  qualifications  are  listed  below,  and  the  results  of 
their  evaluations  follow. 

Evaluator  Qualifications 

John  Milanski  is  a  Staff  Human  Factors  Engineer  with  eleven  years  of  experience  conducting 
usability,  human  factors,  and  programming  projects  using  methods  such  as  heuristic  evaluation, 
usability  testing,  rapid  ethnography,  and  rapid  prototyping.  Mr.  Milanski  has  a  B.S.  in  Aerospace 
Engineering  and  Masters  in  Human-Centered  Product  Design  with  a  focus  on  field  research. 

As  a  cultural  anthropologist  at  Alion,  Ursula  Lauper  is  responsible  for  conducting  user-centered 
research  and  usability  analyses  for  product  development.  Recently,  she  has  performed  field 
research  and  product  evaluations  for  DARPA  and  the  U.S.  Army  Research  Lab.  Ms.  Lauper  has 
a  Masters  in  Cultural  Anthropology. 

Dr.  Lila  Laux  is  a  Senior  Cognitive  Engineer  at  Alion  with  over  25  years  of  experience  in  the 
fields  of  human  factors  and  psychology.  In  addition  to  eight  years  as  a  professor,  Dr.  Laux  has 
focused  on  developing  and  practicing  methods  to  assess  individual  skills  and  predict 
performance.  Dr.  Laux  has  degrees  in  Biology,  Applied  Psychology,  and  Human  Factors. 
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Results  of  Heuristic  Interface  Evaluation 


Heuristics  Violated  Index 

1  =  Match  system-real  world  (familiar  words,  intuitive  layout) 

2  =  Consistent  interface/follows  interface  standards  (words,  objects,  operations,  behaviors) 

3  =  Visible  system  status  (provides  constant  feedback) 

4  =  Error  preventions 

5  =  Error  recovery  (clear  messages) 

6  =  User  Control  (provides  exits/undos) 

7  =  Visual  clues  for  user  action 

8  =  Aesthetic s/minimalistic  design  (relevant  info,  aligned,  consistent  placement) 

9  =  Recognition  over  using  memory  (don’t  have  to  remember  info) 

10  =  Makes  the  user  smart  (reduce  the  need  for  memory,  calculating  relation) 

11=  Flexibility/efficiency  of  use  (novice/expert  use,  tailoring) 

12  =  Help/documentation  (task  focused,  not  too  large,  easy  to  search) 

Severity  Rating  Index 

1  =  Frequency  (H,  M.,  L) 

2  =  Impact  (H,  M,  L) 

3  =  Persistence/Difficulty  to  work-around  (H,  M.,  L) 

4  =  Overall  Severity:  l=Severe,  2=Major,  3=Minor,  4=Cosmetic 
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Screen 


Evaluator 


Issue 


Significance 


Heuristics 

Violated 


2D-Altitude 

Indicator 

Milanski 

BUG:  Not  sure  the  displayed  values  are  correct 

2D  -  Data  to  plot 
selection  bar 

Lauper 

Selected  boxes  don’t  refresh  to  default  when  a  new 
scenario  is  run 

If  user  selects  an  assortment  of  graph  types  to  view 
(whether  or  not  Reset  is  selected),  the  boxes  stay 
checked  if  a  new  scenario  is  started.  Is  this  the 
behavior  that  users  would  expect? 

2D  -  Event 
Sequence  1  -4  y- 
axis  on  illusion 
Flags  tab 

Lauper 

How  is  Event  defined?  An  SDAT  event  is  a  “step”  in  a 
disorientation  illusion.  For  example,  the  1st  Leans  event 
is  a  sub-threshold  roll  of  a  certain  amount  for  a  certain 
duration 

What’s  the  difference  between  the  4  events?  Why 
will  things  show  up  at  different  (discontinuous) 
times  under  the  same  event 

Severity  Rating 

12  3  4 

h  h  h  1 


1  1  m  4 

h  m  m  2 


Suggestion 
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Significance 

Heuristics 

Violated 

Severity  Rating 
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2 

3 

4 

2D  -  illusion 

65 

flags  tab 

Milanski 

A  grid  would  make  it  easier  to  interpret  results 

1 

m 

1 

4 

66 

2D  -  illusion 
flags  tab 

Milanski 

Illusion  Event  graph  should  have  the  same  length  x-axis 
as  the  Results  Graphs,  rather  than  stopping  after  the  last 
observed  illusion  event 

m 

m 
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3 
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2D-illusion  flags 
tab 

Milanski 

Unclear  what  ‘event’  means  in  this  context  -  is  a  timeline 
created  for  every  SD  event? 
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flags  tab 

Lauper 

Coriolis  scenario  doesn’t  trigger  an  illusion  event 

Is  it  only  picking  up  Leans  for  now? 
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2D  -  illusion 
flags  tab 

Milanski 

Illusion  Flags  are  not  being  generated  when  illusion  Type 
is  set  to  ‘All’ 
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2D  -  Results 
Graph 

Milanski 

Would  be  helpful  to  see  a  history  of  the  countermeasures 
employed  and  their  type 

1 

1 
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2D  -  Results 
Graph 

Lauper 

If  all  graph  options  have  been  deselected,  the  original  set 
will  continue  to  run  while  the  selection  boxes  remain 
empty 

Should  the  check  boxes  automatically  repopulate  to 
reflect  the  current  graphs  if  a  selection  error  (e.g., 
deselection  all  graphs)  is  made? 
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2D  -  Results 
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Graph 

Milanski 

A  grid  would  make  it  easier  to  interpret  plots 
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2D  -  Results 
Graph 

Milanski 

Hidden  Feature:  Had  no  idea  you  could  pause  the  cursor 
over  a  point  on  the  graph  and  the  x-y  value  would  appear 
in  a  tooltip 
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2D  -  Results 
Graph 

Milanski 

Don’t  know  you  can  select  4  data  series  until  you  high 
the  REFRESH  button 
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Change  label  (‘Select  up  to  4’) 

14 

2D  -  Results 
Graph 

Milanski 
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Extra  clicks:  Why  must  the  user  hit  the  REFRESH 
button? 

Clicks  are  precious.  Removing  clicks  makes  the  UI 
easier  and  faster  to  use.  Isn’t  this  really  a  button 
that  means  ‘Display’  currently  selected  graphs? 

(Lila) 
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2D  -  Results 
Graph 
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No  way  for  users  to  know  they  can  only  select  a 
minimum  of  1 /maximum  of  4  graphs 
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Graph 
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The  dynamic  y-axis  on  the  time  plots  can  make 
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2D  -  Results 
Graph 
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Scale  used  for  the  actual  vs.  perceived  in  some  instances. 
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99 

2D  -  Results 
Graph 
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Graph 
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simulation  icon 
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Lauper 
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h  m  m  2 


All  other  tabs  clear  their  data  except  this  one  m  m  h  2 

Floating  and  Tabbed  Document  don’t  become 

available  until  Dockable  is  selected  m  m  m  2 


At  first  glance  I  thought  it  was  a  typo  for  “Stop”. 

Maybe  “Advance”?  1114 


All  -  Abort  Lauper 

Simulation 

48  Laux 


Unlike  the  other  control  bar  features  (play,  step, 

No  icon  for  this  function  on  control  bar.  Also,  no  halt,  etc.)  the  only  way  to  access  Abort  is  via 

description  in  the  manual  (Laux)  Execution  on  the  menu  bar 


mm3 


All  -  Blue  hot 
link  text 

43 

Lauper 

Inconsistent  functionality 

Clicking  Select  Flight  Data  File  acts  as  a  check  to 
make  sure  the  file  is  valid;  clicking  some  other 
labels  brings  up  the  contextual  user  guide;  clicking 
till  others  does  nothing.  I’m  assuming  most  text 
will  eventually  be  linked  to  the  user  manual,  but 
that  till  means  that  Select  Flight  Data  File  acts 
differently  from  the  others 

All  -  Halt  and 
Abort  Simulation 
45  Labels 

Lauper 

unclear  of  the  difference  between  them?  Abort  cancels 
the  sim  and  collects  no  data.  Halt  stops  the  sim  but  can 

m  1  1  3 

m  m  h  2 


Suggestion 


Saint  Issue 


SAINT  ISSUE 


123 


Issue 


Screen 


Evaluator 


Issue 


Significance 


Heuristics 

Violated 


begin  the  sim  where  it  left  off. 


All  -  Halt  to  Step  Lauper 
simulation 


50 


Laux 


Since  the  simulation  isn’t  starting  over  from  the 

Changing  from  Halt  to  Step  mode  clears  out  the  Results  beginning  I  would  expect  the  Results  Graph  to 
Graph  tab  continue  where  they  left  off.  I  agree  (Lila) 


All  -  Input 
75 


Laux 


Cannot  use  the  Enter  key  on  all  windows  where  there  is 

an  OK  button  -  this  is  non-standard  aggravating 


2,  6,  1 1 


All  -Main 

18 


BUG:  The  app  seems  to  crash  when  a  data  set  is  fun  in 
Milanski  full  screen  mode 


All  -  Main 
Screen 


5 


The  order  should  convey  some  information,  e.g., 
order  of  steps,  importance,  rather  than  just  being 
random.  I  don’t  think  the  order  is  “random”  I  think 
they  are  in  the  order  that  you  have  asked  to  “View” 
them  from  the  View  Menu  -  I  noticed  the  order 

Milanski  changed  every  time  I  hid  them  and  then 

Inconsistent:  Default  order  of  the  tabs  (Analysis-  reinstantiated  them  from  the  View  menu.  This 

Illusion-Results-Animator-Counter)  does  not  match  the  creates  inconsistency,  but  I  don’t  know  what  to  do 
Laux  order  in  the  View-Windows  drop  menu  (Alphabetical)  about  it. . .  (Lila)  4, 7, 8, 9,  10 


All  -  Main 

§creen  Non-Standard:  Can’t  access  HELP  using  the  menus, 

12  Milanski  only  through  hyperlinks 


2,  12 


All  -  Main 

gcreen  The  tabs  are  not  equivalent  but  they  are  presented  that 

67  Milanski  way,  i.e.,  one  is  setup,  the  others  are  results 


All  -  Main 
Screen 


1 


BUG:  When  trying  to  expand  the  screen  using  the  upper 
right  corner,  an  error  was  generated  (“Unhandled 
exception  has  occurred  in  a  component  in  your 
Milanski  application...”) 


5 


All  -  Main 
Screen  - 
Textboxes  on 
Analysis  Setup 


97 


Laux 


The  red  circle  with  an  exclamation  point  that  instantiates 
upon  unacceptable  entry  into  the  text  field  is  non¬ 
standard  and  there  is  no  explanation  of  what  it  is  or  that 
the  reason  the  app  doesn’t  run  is  because  you  have  an 
error  here.  It  seems  like  it  the  user  is  being  warned  rather 
than  given  an  error  message  (warning  means  that  there  is 

a  serious  hazard).  The  roll  over  explanation  is  good,  but  When  the  app  doesn’t  run  after  the  “Begin 

the  user  has  to  see  the  symbol  and  roll  over  it.  There  is  Simulation  “  icon  is  pressed,  the  user  may  not  know 

no  explanation  of  the  icon  in  the  User’s  Guide  why  and  may  not  “get”  the  red  circle  icon  2,  5,  7,  9,  10,  12 


All  -  Main 

Screen  View  Unnecessary  complexity:  View-Layout  drop  menu  has 

6  Milanski  only  one  option  and  it  is  related  to  View-Windows 


Severity  Rating 

12  3  4 

m  m  m  2 

m  1  1  3 

m  h  m  2 

1114 

m  1  1  3 

h  1  1  3 

h  h  m  1 


1  1  m  3 

h  1  m  3 


Suggestion 


Allow  Help  to  be  accessed  from  Help 
menu 

Separate  the  setup  tab  from  results  tab 
with  a  space  or  blank  tab? 


Explain  the  icon  in  the  manual  at  the 
point  where  it  could  appear. 

View- Windows  and  View-Layout 
could  be  combined  into  one  list  under 


124 


25 

59 

60 

61 

62 

2 

47 

38 

39 

49 

58 


Screen  Evaluator 


Issue 


Significance 


Heuristics 

Violated 


Severity  Rating 

12  3  4 

View 


Suggestion 


All  -  Main 

Simulation  Speed  Drop  list  next  to  speed  icon  does  not  indicate  the  current 

Milanski  simulation  speed 


3 


All  -  Menu  - 

Open  function  File-Open  only  looks  for  saint  files,  whereas  ‘select  flight  Will  the  user  understand  that  ‘open’  and  ‘select  file’ 

Lauper  data  file’  wants  a  .xls  file  are  looking  for  different  things 


h  1  h  3  SAINT  ISSUE 

m  h  m  2 


All  -  Menu  bar 

Milanski  Why  isn’t  there  an  Autosave  setting 


h  1  3 


All  -  Menu  bar 

Milanski 


Field  validation  should  remove  the  need  to  “Check  for 
errors”? 


All  -  Menu  Bar  - 

Execution  -  Play  Why  does  Pause-Play  erase  data  points  prior  to  the  pause 

Milanski  but  Pause  -  Resume  does  not? 


h  1  1  3 


m  h  h  2 


All  -  Menu  Bar  - 
View  -  Windows 

Milanski 

Inconsistent  with  Animator  3D  icons.  Expected  that 
icons  to  left  of  listed  windows  would  indicate  that  the 
window  was  opened  or  closed,  like  the  icons  do  for 
Animator  3D  -  Show  Floor  Grid  (which  provides  good 
user  feedback) 

All  -  Pause  vs. 
(re)play  function 

Lauper 

Pressing  Play  after  Pausing  deletes  previous  lines  in 
Results  Graph  (toggling  Pause  on  and  off  works  fine) 

Is  there  any  way  to  keep  restart  from  where  it  left 
off?  I  noticed  in  the  user  guide  that  this  is 
intentional  but  I  don’t  understand  why.  Will  the 
users?  I  agree  (Lila) 

All  -  Sections 
sliding  window 

Lauper 

Laux 

Unnecessarily  redundant.  What  utility  does  this  have? 

Users  can  already  select  screen  sections  via  the  tabs 
on  the  main  window  or  through  View  on  the  menu 
bar.  Why  do  they  need  a  3rd  method,  especially 
since  it’s  located  right  next  to  the  tabs? 

All  -  Sections 
sliding  window 

Lauper 

Order  of  sections  doesn’t  match  tabs 

Tabs  list  the  sections  in  a  different  order  from  the 
Sections  slider 

h  1  1  3 


h  h  h  1  SAINT  ISSUE 


m  1  1  3 


m  1  m  3 


All  -  Simulation 
feature 

Lauper 

Double-click  is  needed  for  function  to  work 

Seems  to  be  working  as  an  on/off  toggle,  which 
doesn’t  match  the  functionality.  If  the  purpose  of 
the  feature  is  to  advance  the  simulation  by  x%,  then 
each  click  should  do  just  that.  Having  an  “off’ 
mode  for  something  that  doesn’t  continue  on  its 
own  doesn’t  make  any  sense 

All  -  Simulation 
speed  indication 

Lauper 

There’s  no  indication  on  the  interface  that  tells  the  user 
what  speed  the  simulation  will  run/is  running  at 

I  see  it  hasn’t  been  set  up  in  Execution  settings  yet. 
Depending  on  how  users  play  with  this,  they  might 

m  m  m  2  SAINT  ISSUE 


1114  SAINT  ISSUE 


125 


Issue 


Screen 


Evaluator 


Issue 


Significance 


want  a  “speed”  display  that’s  continually  visible 
when  they’re  running  simulations 


..  .  ..  Severity  Rating 

Heuristics 

Violated 

12  3  4 


All  -  Simulation 
Speed  Selection 

57  Lauper 


The  drop-down  speed  selection  menu  has  no  check  mark, 
so  the  user  gets  no  feedback  to  indicate  which  is 

currently  selected,  and  if  a  new  selection  has  registered  Check  mark  should  appear  next  to  the  current  speed 


n  h  h  2 


All  -  View  Menu 
Bar 


42 


Lauper 


No  way  to  re-open  individual  dockable  windows 
Sections,  Output,  and  Attitude  Indicator  after  they’ve 
been  closed  out  except  to  reset  the  entire  interface. 


E.g.,  my  first  instinct  is  to  click  the  X  on  the 
Sections  slider  to  close  it  (instead  of  waiting  for  it 
to  close).  This  removes  the  slider  from  the 
interface.  The  only  way  to  get  it  back  is  to  select 
View-Layout=Reset  Screen  Layout.  What  about 
making  each  of  the  dockable  windows  available  for 
reopening  individually  through  the  same  menu? 


n  h  2 


All  -  View 
Windows  menu 
bar 

41  Lauper 


Naming  inconsistent  with  Sections  slider 


Are  these  “windows”  or 

“sections”?  If  we’re  keeping  both  navigation 

methods  the  naming  should  be  consistent 


1  m  1  4 


All  Windows 


93 


Laux 


Analysis  setup 

68  Milanski 


There  is  really  only  one  window  in  this  application  by 
Windows  standards  -  the  Main  Window  (which  is  a 
frame  with  the  panes  in  it).  All  of  the  other  parts  of  the 
window  should  be  referred  to  (especially  in  the  manual) 
as  panes  (viewing  areas  within  windows) 


Should  sample  rate  be  in  terms  of  hz  or  sample/sec 


1,2,  12 


h  1  1  3 

1114 


Analysis  setup 


Milanski 


Non-standard  use  of  dimming:  Before  an  analysis  is 
done,  the  Recovery  -  Coriolis  -  Graveyard  -  Leans  tabs 
are  grayed  out,  but  still  able  to  activate  those  tabs. 
Usually  grayed  -out  indicated  that  the  feature  is 
unavailable/can’t  be  selected.  Also  differs  from  the  way 
the  main  tabs  are  displayed.  (Analysis  -  illusion  - 
Results...) 


2,7 


h  1  1  3 


Analysis  Setup 

7  Milanski 


Non-Standard  Checkboxes:  Until  the  SW  allows  for 
checking  of  multiple  illusion  Types.  The  selection 
should  be  radio  buttons. 


h  1  1  3 


Analysis  Setup 


Milanski 


Laux 


Inconsistent  hyperlinks:  Some  underlined  words  link  to 
help,  others  do  not 


Too  many  proposed  hyperlinks  anyway  -  should 
only  allow  linking  to  “higher”  levels,  not  to  every 
field  in  a  pane  (Lila)  2 


h  1  m  3 


Analysis  Setup 

9  Milanski 


“2  decimal  places”  appears  to  indicate  that  this  is  the 
default,  but  it  is  not  labeled  as  in  other  areas 


2,6 


h  1  m  3 


Suggestion 


SAINT  ISSUE 


SAINT  ISSUE 


Radio  buttons 


126 


Issue 

Screen 

Evaluator 

Issue 

Significance 

Heuristics 

Violated 

Severity  Rating 

Suggestion 

1 

2 

3 

4 

11 

Analysis  Setup 

Milanski 

Non-Standard  checkboxes:  Washout  function  choices 
should  be  checkboxes  if  only  one  can  be  chosen 

Actually,  Washout  function  choices  should  be  radio 
buttons  if  only  one  can  be  chosen  (Lila) 

h 

1 

1 

3 

33 

Analysis  Setup 

Milanski 

Laux 

When  model  is  running,  all  fields  on  this  screen  remain 
active 

Implies  it  is  possible  to  change  settings  and  effect 
results  while  model  is  running 

h 

m 

1 

3 

Should  be  grayed  out  when  the  model  is 
running 

34 

Analysis  Setup 

Milanski 

Program  does  not  display  the  actual  data  sampling  rate 
used  (e.g.,  user  specifies  8  Hz  but  the  data  file  only  goes 
down  to  4Hz  and  that  is  what  the  program  runs  at) 

m 

m 

1 

3 

Prescan  the  file  when  it  is  selected,  and 
place  a  limit  on  data  sampling  rate 
equal  to  that  in  the  file. 

35 

Analysis  Setup 

Milanski 

Is  it  possible  to  use  sliders  for  acceptable  values  in  all 
setup  fields  along  with  the  text  entry  box? 

h 

1 

1 

3 

94 

Analysis  Setup 

Laux 

Data  rounding  field  -  if  enter  a  lot  of  numbers  it  presents 
the  red  error  circle  (see  above),  even  when  you  have 
presented  an  integer  that  “fits”  in  the  field.  The  roll  over 
message  says:  “Not  a  valid  entry  must  be  set  to  an 
integer  value”  which  does  not  tell  you  what  the  problem 
is.  And  the  red  circle  doesn’t  go  away  when  you  replace 
whatever  you  have  put  in  there  with  2  (which  is  the 
default  value)  until  you  go  to  another  field 

1,3,  4,  5,  7,  10, 
12 

m 

m 

1 

3 

10 

Analysis  Setup 

Milanski 

Data  Rounding  label  gives  no  clues  of  the  values  allowed 

4 

h 

m 

m 

2 

95 

Analysis  Setup  - 
Countermeasures 

Laux 

Misspelled  demonstrate 

1 

1 

1 

4 

26 

Analysis  Setup  - 
Countermeasures 

Milanski 

Doesn’t  show  the  possible  values  for  each  field 

h 

1 

1 

3 

28 

Analysis  Setup  - 
Countermeasures 

Milanski 

No  feedback  about  how  the  algorithm  determines 
countermeasure 

3 

h 

1 

h 

3 

Provide  a  visual  link  between  the  cause 
(e.g.,  disorientation  type)  to  the  effect 
(countermeasure) 

89 

Analysis  Setup  - 
Create  New 
Analysis 

Laux 

Data  Sampling  Rate  -  very  opaque  to  the  user.  What  are 
they  able  to  select?  What  happens  if  they  put  in  a  value 
that  is  very  unrealistic?  It  lets  you  enter  ANY  value 

3 

96 

Analysis  Setup  - 
Output 

Laux 

What  is  the  unit  for  duration  and  why  does  it  have  12 
decimal  places  (if  this  is  seconds,  that  is  a  VERY  small 
amount  of  time) 

2,7 

3 

2  dec  places  max  and  unit  (sec) 

90 

Analysis  Setup  - 
Reset  to  Defaults 
buttons 

Laux 

These  buttons  typically  belong  to  panes  in  the  Main 
window,  but  it  is  not  necessarily  clear  which  fields  they 
relate  to.  Standard  Windows  design  would  put  them 

3 

127 


Issue 

Screen 

Evaluator 

Issue 

Significance 

Heuristics 

Violated 

Severity  Rating 

Suggestion 

1 

2 

3 

4 

inside  a  Group  box  with  all  of  the  controls  they  relate  to, 
but  here  the  panes  are  not  clearly  delineated  and  the 
controls  are  not  grouped. 

55 

Analysis  Setup  - 
Select  Illusion 
Type 

Lauper 

Laux 

Leans  is  the  default  instead  of  all 

I’d  expect  All  to  be  the  default.  I  would  not  expect 
any  default  unless  you  have  checked  with  users  and 
determined  from  them  which  one  they  will  use  most 
of  the  time  (Lila). 

h 

h 

1 

3 

36 

Analysis  Setup  - 
Vestibular 

Milanski 

Laux 

The  tab  order  seems  random.  The  tab  order  on  all  of  the 
tabs  and  windows  seems  random  (Lila) 

3 

make  the  tab  order  logical  or  take  out 
tab  ability 

Animato  3D 

77 

Settings  Tab 

Laux 

Should  be  disabled  because  it  isn’t  functional 

4 

20 

Animator  3D 

Milanski 

Hidden  feature:  No  indication  that  the  viewpoint  can  be 
zoomed  using  the  mouse  scroll  wheel 

1 

m 

1 

4 

22 

Animator  3D 

Milanski 

What  does  the  axes  relate  to?  Mag  north?  Maybe  should 
be  relabeled  ’pitch,  roll,  yaw”  rather  than  “x,y,z” 

1.2 

1 

1 

h 

4 

72 

Animator  3D 

Milanski 

Unclear  what  ‘Exceeds  Threshold” 

refers  to  (pitch,  roll,  yaw) 

1 

1 

1 

4 

Animator  3D 

19 

Milanski 

Possible  to  drag  the  axes/planes  off  screen  where  they  are 
“lost” 

1 

m 

h 

3/4 

17 

Animator  3D 

Milanski 

Hidden  feature:  No  indication  that  the  viewpoint  can  be 
changed  by  dragging  the  axes. 

1 

m 

m 

3 

23 

Animator  3D 

Milanski 

What  is  the  difference  between  the  three  ‘velocity’  labels 

4,9 

h 

1 

1 

3 

Change  label  to  ‘pitch  velocity’  etc. 

64 

Animator  3D 

Milanski 

BUG:  At  the  end  of  an  800%  speed  run,  while  on  the 
Animator  3D  screen,  a  message  popped  up  saying 
“Animator  3D  is  currently  in  Execution  mode.  Do  you 
want  to  switch  to  edit  mode?” 

1 

m 

m 

3 

21 

Animator  3D 

Milanski 

Hard  to  compare  data:  the  main  function  seems  to  be  to 
compare  actual  to  perceived,  but  the  data  is  separated  on 
screen  making  comparison  harder. 

h 

m 

h 

2 

Place  data  in  four  columns:  labels, 
actual,  perceived,  and  plane  view. 
“Plane  view”  means  a  top  down  view 
of  the  two  aircraft  icons  overlaid  or 
very  close  to  each  other. 

128 


Issue 


Screen 


Evaluator 


Issue 


User  Guide 

63  Milanski 

User  Guide 

7 1  Milanski 

User  Guide 


76 

Laux 

User  Guide 

78 

Laux 

User  Guide 

80 

Laux 

User  Guide 

81 

Laux 

User  Guide 

83 

Laux 

User  Guide 

84 

Laux 

User  Guide 

88 

Laux 

User  Guide 

91 

Laux 

User  Guide 

92 

Laux 

User  Guide 

79 

Laux 

User  Guide 

82 

Laux 

User  Guide 

85 

Laux 

User  Guide  says  that  the  simulation  speed  cannot  be 
changed  during  playback  (pg  19),  but  it  seems  to  work 


p.21,  Ch4  section  describing  ‘results  graphs’  is 
mistakenly  labeled  ‘analysis  window’ 


Typos:  p.57,  os  should  be  is;  p65  Base  should  be  based; 
certainly  frequently  misspelled  in  manual 


p30  discussion  of  Task  description  windows/document 
windows  no  correct  for  SDAT 


p  45  refers  to  a  red  ‘x’  button  that  can  be  used  to  close 
the  window  -  there  is  none 


p  47  -  Where  is  the  Utilities  menu?  Simulation  speed  is 
not  under  a  Utilities  menu 


p  55  There  is  no  “load”  button.  The  button  says  “Open”. 
Load  is  preferable 


p.55  refer  the  user  to  the  sections  that  explain  the 
different  illusion  types  later  in  the  manual 


p  57  next  to  last  line-simultaneously  with  what? 


Footnotes  reverences  missing 


p.  74  Event  4  Vertical  Speed  definition  seems  the  same 
as  the  definition  for  Event  4  Roll  Angle 


Hide  and  Close  window  mean  the  same  thing  -  should 
use  one  term  unless  they  actually  produce  different 
results 


p.  47  There  is  no  Analysis  Window  (this  is  referred  to 
multiple  times  in  the  manual) 

p.  56  out  of  sync  with  the  pane  which  has  4  choices  (it 
adds  “All”) 


Significance 


Heuristics 

Violated 


Severity  Rating 


Suggestion 


1  2  3 


1  1  1 


1  1  1 


1  1  1 


Take  this  out 


remove  “red” 


change  manual 


correct  this 


4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

3 

3 

3 


129 


Appendix  C.  Patent  Application 


UTILITY  APPLICATION 
BY 

RONALD  L.  SMALL 
JOHN  W.  KELLER 
ALIA  M.  FISHER 
CHRISTOPHER  D.  WICKENS 


FOR 

UNITED  STATES  PATENT 
ON 

METHOD  FOR  SPATIAL  DISORIENTATION  IDENTIFICATION,  COUNTERMEASURES, 

AND  ANALYSIS 


Docket  No:  437728 
Sheets  of  Drawings:  Six  (6) 

Express  Mail  No.:  EV  498086185  US 


Micro  Analysis  and  Design 
4949  Pearl  East  Circle,  Suite  300 
Boulder,  Colorado  80301 

Attorneys 
Daniel  W.  Roberts 
David  Lee 


130 


METHOD  FOR  SPATIAL  DISORIENTATION  IDENTIFICATION,  COUNTERMEASURES, 

AND  ANALYSIS 

RELATED  APPLICATION 

[0001]  This  application  claims  priority  of  U.S.  Provisional  Application  Serial  Number  60/678,919, 
filed  on  May  6,  2005. 

GOVERNMENT  RIGHTS 

[0002]  The  U.S.  Government  has  a  paid-up  license  in  this  invention  and  the  right  in  limited 
circumstances  to  require  the  patent  owner  to  license  others  on  reasonable  terms  as  provided  for  by  the 
terms  of  Contract  FA8640-04-C-6457,  awarded  by  USAF/AFMC,  Air  Force  Research  Laboratory,  2310 
Eighth  Street,  Building  167,  Wright-Patterson  AFB,  Ohio  45433-7801. 

FIELD  OF  THE  INVENTION 

[0003]  The  invention  relates  generally  to  the  field  of  human  spatial  disorientation  (SD).  It  addresses 
the  problems  of  identifying,  correcting,  and  analyzing  SD  episodes  as  experienced  by  human  subjects  in 
environments  where  spatial  disorientation  may  occur,  such  as,  for  example,  an  aircraft  pilot  in  an  aircraft. 


BACKGROUND 

[0004]  Spatial  disorientation  (SD)  is  the  mistaken  perception  of  a  person's  position,  attitude  and  motion 
relative  to  the  earth  or  significant  objects  visible  to  the  person,  such  as,  for  example,  mountains,  trees, 
buildings  or  the  like. 

[0005]  Spatial  disorientation  is  a  normal  human  response  to  accelerations  in  flight,  and  has  been 
recognized  since  the  early  days  of  flight.  The  cost  of  SD  to  the  U.S.  military  is  over  $300  million  per 
year,  with  comparable  costs  to  U.S.  civil  aviation.  Without  question,  SD  contributes  more  to  the  cause  of 
aircraft  accidents  (civilian  as  well  as  military)  than  any  other  physiological  problem  in  flight.  Regardless 
of  flight  time  experience,  all  aircrew  members  are  subject  to  disorientation. 

[0006]  Despite  significantly  increased  research  over  the  past  decade,  the  rate  of  accidents  caused  by 
SD  has  not  decreased.  With  few  exceptions,  the  most  recent  research  emphases  have  been  limited  to 
understanding  the  physiology  of  SD.  The  new  knowledge  gained  by  such  research  has  not  been  translated 
into  tools  (e.g.,  training,  displays,  automation)  that  help  pilots  avoid  SD  and  minimize  its  effects  if  it  does 
occur.  Although  a  very  common  experience  for  pilots  in  aircraft,  SD  events  and  their  associated  problems 
may  also  be  experienced  by  boat  operators,  divers,  astronauts,  firefighters  and  other  persons  in 
environments  where  visual  cues  may  or  may  not  agree  with  the  perceived  feelings  of  motion.  Simply 
stated,  if  a  person  is  in  an  environment  with  low  visibility  and  impaired  attitude  awareness,  there  is  an 
elevated  chance  they  will  experience  an  SD  event. 

[0007]  There  are  three  types  of  SD,  Type  1  -  unrecognized.  Type  2  -  recognized  and  Type  3  - 
incapacitating,  each  of  which  is  most  commonly  referred  to  with  respect  to  aircraft  pilots.  With  Type  1, 
the  pilot  is  not  aware  of  critical  control  or  flight  parameters  of  the  aircraft,  and  therefore  may  control  the 
aircraft  with  erroneous  assumptions.  With  Type  2,  the  pilot  perceives  a  problem  (resulting  from  SD)  but 
fails  to  recognize  it  as  SD.  Typically  the  pilot  will  believe  that  the  aircraft  instruments  are 
malfunctioning.  With  Type  3,  the  pilot  experiences  such  an  overwhelming  sensation  of  movement  that  he 
or  she  can  not  reorient  himself  or  herself  by  using  visual  cues  or  the  aircraft  instruments. 

[0008]  A  human  being's  perception  of  motion  is  a  result  of  the  vestibular  system,  otherwise  known  as 
the  inner  ear.  The  vestibular  system  (organ  of  equilibrium),  consists  of  two  structures  -  1)  the 
semicircular  canals,  which  detect  changes  in  angular  acceleration;  and  2)  the  otolith  organs  (utricle  and 
saccule),  which  detect  changes  in  linear  acceleration  and  gravity. 

[0009]  The  semicircular  canals  are  three  half-circular,  interconnected  tubes  in  three  planes 
perpendicular  to  each  other.  Each  plane  corresponds  generally  to  rolling,  pitching  or  yawing  motions. 
Although  there  are  two  systems  (one  for  each  ear),  they  are  collectively  in  operation  as  one  system.  Each 
canal  is  filled  with  a  fluid  called  endolymph.  A  motion  sensor  is  provided  in  the  form  of  cupula  (a 
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gelatinous  structure)  and  hairs  extending  from  hair  cells  below  the  cupula;  the  ends  of  the  hairs  are 
embedded  in  the  cupula.  The  cupula  and  the  hairs  move  as  the  fluid  inside  the  canal  moves  in  response  to 
an  angular  acceleration. 

[0010]  The  otolith  organs,  the  saccule  and  utricle,  are  located  in  each  ear  and  are  set  at  right  angles  to 
each  other.  The  utricle  detects  changes  in  linear  acceleration  in  the  horizontal  plane,  while  the  saccule 
detects  gravity  changes  in  the  vertical  plane.  Inertial  forces  resulting  from  linear  accelerations  cannot  be 
distinguished  from  the  force  of  gravity. 

[0011]  As  the  issue  of  SD  can  lead  to  loss  of  life  and  the  destruction  of  property,  several  prior  art 
methods  have  been  developed  in  various  efforts  to  address  SD  with  respect  to  aircraft  pilots.  This  art 
includes  flight  simulation  devices  such  as  described  in  U.S.  Pat.  No.  4,710,128  to  Wachsmuth  et  al., 
which  provides  a  controlled  environment  for  creating  SD  events.  However,  certain  SD  events  arise  from 
conditions  below  human  perception  thresholds. 

[0012]  Also  in  the  related  art  is  U.S.  Pat.  No.  5,285,685  to  Chelette,  which  provides  a  method  and 
apparatus  for  communicating  perceived  attitude  information  from  a  test  subject.  Again,  although  perhaps 
beneficial  for  some  instances  of  SD,  there  are  forms  of  SD  which  are  below  perception  and  others  that  are 
so  overwhelming  that  the  subject  loses  all  perception  of  perceived  attitude. 

[0013]  U.S.  Pat.  No.  5,629,848  to  Repperger  et  al.,  is  focused  upon  an  SD  detector  system  capable  of 
warning  a  pilot  of  potentially  disorienting  flight  conditions  in  response  to  a  Kalman  filter  modeling  of 
human  response  characteristics.  More  specifically,  a  Kalman  apparatus  produces  a  state  estimate  of  both 
the  true  position  and  orientation,  as  well  as  the  pilot's  perceived  position  and  orientation  of  the  aircraft. 
[0014]  The  Repperger  system  is  based  in  part  on  both  the  otolith  and  semicircular  canal  responses  of 
human  physiology.  The  Repperger  system  is  an  on-board  only  system,  active  only  during  flight,  and  its 
function  is  to  determine  only  when  an  SD  event  is  or  is  not  occurring.  When  an  error  of  sufficient 
magnitude  occurs  between  the  Kalman  filter's  true  value  and  perceived  estimate,  an  SD  event  is  deemed 
to  be  occurring  and  a  visual  warning  is  provided  to  the  pilot. 

[0015]  Repperger  does  not  consider  the  developing  probability  of  the  SD  event  or  the  type  of  SD 
event.  In  addition,  Repperger  does  not  provide  for  post-event  analysis  and/or  comparison  to  other  similar 
events.  Further  still,  Repperger  does  not  consider  either  a  range  of  warnings  or  different  methods  and/or 
types  of  delivery  selected  for  the  best  chance  of  reaching  the  subject  pilot  and  helping  him  or  her 
overcome  the  SD  event. 

[0016]  Hence,  there  is  a  need  for  a  spatial  disorientation  identification  method  and  system  that 
overcomes  one  or  more  of  the  technical  drawbacks  identified  above. 

SUMMARY 

[0017]  The  present  disclosure  advances  the  art  by  providing  a  system  and  method  for  spatial 
disorientation  identification,  countermeasures  and  analysis. 

[0018]  In  particular,  and  by  way  of  example  only,  according  to  an  embodiment,  provided  is  a 
computer-readable  medium  on  which  is  stored  a  computer  program  for  detecting,  analyzing  and 
responding  to  a  spatial  disorientation  event.  The  computer  program  includes  an  input  routine  operatively 
associated  with  an  input  device  for  receiving  real  time  data,  recorded  data,  subject  preference  information 
or  combinations  thereof.  The  data  include  environment  data  from  an  environment,  the  environment  data 
including  true  position  and  orientation  of  the  environment,  and  subject  data  from  a  subject  within  the 
environment.  A  vestibular  attitude  calculator  routine  computes  the  perceived  subject  attitude  of  the 
subject  within  the  environment  based  on  the  environment  data  and  subject  data.  The  vestibular  attitude 
calculator  includes  a  Washout  routine  to  calculate  a  Washout  value;  a  vestibular  illusion  routine  to 
calculate  the  probability  of  a  vestibular  illusion;  a  threshold  adjustment  routine  permitting  adjustment  of 
Washout  thresholds  and  vestibular  thresholds  based  on  provided  subject  preference  information;  a 
countermeasure  routine  operating  in  response  to  the  Washout  value  and  the  probability  of  a  vestibular 
illusion,  and  an  output  routine  operatively  associated  with  an  output  device  to  provide  the  true  position 
and  orientation  of  the  environment  and  the  perceived  subject  attitude. 


132 


[0019]  In  an  alternative  embodiment,  provided  is  a  method  for  analyzing  a  spatial  disorientation  event 
post-hoc,  including:  collecting  environment  data  elements  recorded  from  an  environment;  collecting 
subject  data  recorded  from  a  subject  within  the  environment;  calculating  perceived  subject  attitude  of  the 
subject  within  the  environment  for  each  environment  data  element;  evaluating  the  environment  data,  the 
subject  data  and  the  perceived  subject  attitude  to  determine  the  probability  of  a  spatial  disorientation 
event,  and  reporting  the  probability  of  the  spatial  disorientation  event. 

[0020]  In  yet  another  alternative  embodiment,  provided  is  a  method  for  combating  spatial 
disorientation,  including:  collecting  real  time  environment  data  from  an  environment;  collecting  real  time 
subject  data  from  a  subject  within  the  environment;  calculating  perceived  subject  attitude  of  the  subject 
within  the  environment  for  each  environment  data  element  and  predicting  Washout;  evaluating  the 
environment  data,  the  subject  data,  the  perceived  subject  attitude  and  Washout  to  determine  the 
probability  of  a  spatial  disorientation  event  and  the  type  of  spatial  disorientation  event;  implementing,  in 
response  to  the  probability  of  a  spatial  disorientation  event,  at  least  one  countermeasure;  and  recording  the 
environmental  data  and  subject  data  as  event  data  for  post-hoc  review.  The  environment  data  include  true 
position  and  orientation  of  the  environment;  the  Washout  evaluated  as  a  non-linear  element,  and  the 
countermeasure  is  selectively  chosen  from  a  group  of  multi  sensory  countermeasures  and  countermeasure 
actions  based  on  the  environment  data  and  spatial  disorientation  probability. 

BRIEF  DESCRIPTION  OF  THE  DRAWINGS 

[0021]  FIG.  1  shows  a  high  level  block  diagram  for  a  spatial  disorientation  identification  system  in 
accordance  with  an  embodiment; 

[0022]  FIG.  2  is  a  graph  illustrating  the  non-linear  property  of  Washout; 

[0023]  FIG.  3  is  a  graph  illustrating  the  Leans  illusion  model  according  to  an  embodiment; 

[0024]  FIG.  4  is  a  graph  illustrating  the  Coriolis  illusion  model  according  to  an  embodiment; 

[0025]  FIG.  5  is  a  graph  illustrating  the  Graveyard  Spiral  illusion  model  according  to  an  embodiment; 

[0026]  FIG.  6  is  a  high  level  flowchart  illustrating  spatial  disorientation  detection  in  real  time  with 
recordation  for  post  hoc  review  according  to  an  embodiment;  and 

[0027]  FIG.  7  is  a  high  level  flowchart  illustrating  spatial  disorientation  detection  in  post  hoc  review. 

DETAILED  DESCRIPTION 

[0028]  Before  proceeding  with  the  detailed  description,  it  is  to  be  appreciated  that  the  present 
teaching  is  by  way  of  example,  not  by  limitation.  The  concepts  herein  are  not  limited  to  use  or 
application  with  a  specific  type  of  system  or  method  for  combating  spatial  disorientation.  Thus,  although 
the  instrumentalities  described  herein  are  for  the  convenience  of  explanation  shown  and  described  with 
respect  to  exemplary  embodiments,  it  will  be  appreciated  that  the  principles  herein  may  be  equally 
applied  in  other  types  of  methods  and  systems  for  detecting  and  combating  spatial  disorientation. 

[0029]  FIG.  1  is  a  high  level  block  diagram  of  the  computer  program  architecture  of  a  spatial 
disorientation  (SD)  system  100  in  accordance  with  at  least  one  embodiment.  SD  system  100  may  be 
implemented  on  a  computer  having  typical  computer  components,  such  as  a  processor,  memory,  storage 
devices  and  input  and  output  devices.  During  operation,  SD  system  100  may  be  maintained  in  active 
memory  for  enhanced  speed  and  efficiency.  In  addition,  in  at  least  one  embodiment,  SD  system  100  may 
be  operated  on  a  computer  network  and  may  utilize  distributed  resources. 

[0030]  To  further  assist  in  the  following  descriptions  and  discussion,  the  following  defined  terms  are 
provided. 

[0031]  "Environment"  -  An  aircraft,  boat,  vehicle  or  other  setting  which  may  provide  conditions  for 
an  SD  event  to  occur.  In  at  least  one  embodiment,  the  Environment  is  an  aircraft. 

[0032]  "Subject"  -  A  person  within  the  environment  subject  to  the  possibility  of  experiencing  an  SD 
event.  In  at  least  one  embodiment,  the  Subject  is  an  aircraft  pilot. 

[0033]  Countermeasure"  -  An  action  taken  to  prevent,  minimize,  or  compensate  for  an  SD  event. 
[0034]  "Washout"  -  The  diminishing  capacity  to  appreciate  continual  motion  at  a  constant  rate. 
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|0035]  "Threshold"  -  The  level  below  which  accelerations  or  motions  are  not  sensed  by  the  human 
vestibular  system. 

[0036]  Threshold  and  Washout  are  significant  factors  in  the  onset  of  SD  events.  Within  the  relevant 
field  of  art,  early  research  suggested  the  typical  Threshold  value  for  all  axes  could  be  Mulder's  constant, 
two  degrees  per  second.  If  a  person  experiences  an  acceleration  of  l°/sec2  for  one  second,  he  or  she  will 
probably  not  sense  that  acceleration  because  the  product  (l°/sec)  is  below  Mulder's  constant.  If  the  same 
acceleration  occurs  for  three  seconds  however,  it  will  likely  be  detected  because  the  product  (3°/sec) 
exceeds  Mulder's  constant.  It  should  also  be  recognized  that  even  a  large  acceleration  of  10°/sec2  is 
unlikely  to  be  felt  if  the  duration  is  less  than  0.2  seconds. 

[0037]  Mulder's  constant,  although  helpful  for  general  purposes  has  now  been  refined  by  Stapleford 
to  reflect  that  the  Threshold  values  are  not  uniform  for  all  axes.  As  refined  by  Stapleford  the  general 
Threshold  values  are  3.2°/sec  for  roll,  2.6°/sec  for  pitch  and  l.l°/sec  for  yaw.  It  is  to  be  appreciated  that 
acceleration  may  be  linear  acceleration  and  or  rotational  acceleration,  and  Washout  can  occur  in  either 
sense. 

[0038]  It  is  further  important  to  note  that  not  all  humans  have  identical  thresholds,  however  Mulder's 
constant  is  a  good  generalization.  It  is  also  important  to  note  that  not  all  acceleration  durations  above 
Mulder's  constant  will  be  sensed.  Distractions,  fatigue  and  other  physiological  reasons  may  exist  to  make 
the  person  oblivious  to  accelerations  and/or  durations  that  exceed  Mulder's  constant. 

[0039]  With  respect  to  Washout,  when  one  or  more  semicircular  canals  of  the  inner  ear  are  put  into 
motion,  the  fluid  within  the  canal  lags  behind  the  accelerated  canal  walls.  Lag  of  the  fluid  is  sensed  by 
hairs  of  the  cupula  and  the  brain  interprets  the  movement  of  the  hairs  as  motion  in  a  direction.  If  motion 
continues  at  a  constant  rate  for  several  seconds  or  longer,  the  fluid  in  the  canals  catches  up  with  the  canal 
walls  and  the  brain  receives  the  false  impression  that  the  turning  has  stopped,  thus  Washout  has  occurred. 
[0040]  It  is  important  to  note  that  Washout  is  non-linear.  More  specifically,  FIG.  2  shows  an 
exponential  decay  curve  that  represents  the  change  in  perceived  rotation  over  time.  In  the  exemplary  case 
the  subject  is  a  pilot  who  has  accelerated  into  a  turn  and  reached  a  5°/sec  yaw  rotation.  At  this  point,  the 
turn  is  held  such  that  the  rate  of  rotation  is  close  to  constant  (i.e.,  there  are  no  above-threshold 
accelerations  for  a  period  of  time).  After  2.5  seconds,  the  sense  of  rotation  is  down  to  about  2°/sec.  With 
respect  to  the  true  5°/sec  yaw  rotation,  this  is  a  3°/sec  difference  between  actual  and  perceived  rotation,  or 
a  Washout  of  about  60%.  Although  described  with  respect  to  yaw,  Washout  may  occur  for  pitch  or  roll, 
as  well. 

[0041]  As  is  further  set  forth  and  described  below,  SD  system  100  is  used  to  identify  and  combat 
spatial  disorientation  in  accordance  with  the  following  primary,  but  not  exclusive,  tenets: 

First  -  the  SD  system  100  acts  to  alert  the  subject  with  a  countermeasure  based  on  the 
probability  of  an  SD  event  occurring,  and  the  greater  the  probability,  the  more  intense 
the  countermeasure. 

Second  -  in  determining  the  probability  of  an  SD  event,  the  SD  system  100  accounts  for 
Washout  of  the  subject,  a  calculation  that  in  at  least  one  embodiment  is  non-linear. 

Third  -  the  selectable  countermeasures  are  multi  sensory. 

Fourth  -  the  SD  system  100  is  capable  of  identifying  a  predicted  SD  event  as  a 
vestibular  illusion  (e.g.,  somatogravic  or  somatogyral  illusion),  and  even  more 
specifically,  identifying  the  specific  type  of  illusion. 

Fifth  -  the  SD  system  100  permits  both  real  time  SD  event  determination  and  post-hoc 
analysis.  With  respect  to  aircraft  flight  for  example,  post  hoc  review  permits  researchers 
to  simulate  flight  patterns  and  identify  commonalities  regarding  SD  events,  thus 
permitting  enhanced  prediction  of  future  SD  events  and  evaluation  of  countermeasures 
applied  to  SD  events. 

[0042]  Returning  to  FIG.  1,  in  at  least  one  high  level  embodiment,  SD  system  100  includes  an  input 
routine  102,  a  countermeasure  routine  106  and  an  output  routine  104,  each  operably  associated  with  a 
Vestibular  Attitude  Calculator  (VAC)  routine  108.  The  VAC  routine  108  further  includes  a  threshold 
adjustment  routine  110,  a  Washout  routine  112  and  a  vestibular  illusion  routine  114. 
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[00431  The  input  routine  102  is  operatively  associated  with  at  least  one  input  device  for  receiving 
real  time  data,  recorded  data,  subject  preference  information  and/or  combinations  thereof.  The  real  time 
or  recorded  data  include  environment  data  from  an  environment  (e.g.,  an  aircraft),  including  true  position 
and  orientation. 

[0044]  The  real  time  or  recorded  data  also  include  subject  data  from  the  subject  within  the 
environment.  In  at  least  one  embodiment  the  received  subject  data  include  vestibular  Threshold  values 
and  Washout  timing  values.  Minimum  rotation  threshold  values  may  also  be  provided.  In  addition,  the 
Threshold  and  Washout  values  may  be  independently  defined  for  each  axis  of  rotation  (yaw,  pitch,  roll), 
based  on  Mulder's  constant,  Stapleford  values,  or  individual  subject  preferences.  Moreover,  the  subject 
may  provide  data  to  modify  the  SD  event  illusion  parameters  so  as  to  increase  the  effectiveness  of  SD 
system  100  in  determining  the  probability  of  an  SD  event.  Subject  data  may  also  include  information 
such  as  the  subject's  name,  date  and  time. 

[0045]  In  addition,  in  at  least  one  embodiment,  subject  data  include  information  indicating  the  true 
position  and  orientation  of  the  subject's  head.  For  example,  the  commercially  available  helmets  worn  by 
fighter  pilots  typically  include  devices  such  as  Polhemus  motion  tracking  system,  accelerometers,  micro¬ 
electrical  mechanical  devices,  or  other  devices  that  accurately  measure  and  report  the  roll,  pitch  and  yaw 
angles  of  the  pilot's  head.  Private  pilots,  boat  operators,  divers,  firefighters,  astronauts  or  other  subjects  in 
other  environments  may  be  easily  fitted  with  commercially  available  accelerometers  or  other  micro¬ 
electrical  mechanical  devices  to  provide  similar  roll,  pitch  and  yaw  angles  of  the  subject's  head.  It  is  to  be 
understood  and  appreciated  that  the  pitch,  yaw  and  roll  axes  of  the  subjects  head  (i.e.,  the  pilot) 
correspond  to  the  same  pitch,  yaw  and  roll  axes  of  the  environment  (i.e.,  the  aircraft).  Further  still, 
subject  data  may  also  include  data  from  the  controls  operable  by  the  subject,  e.g.,  flight  stick  and  foot 
pedals. 

[0046]  Moreover,  in  at  least  one  embodiment,  the  real  time  or  recorded  subject  data  are  gathered 
from  at  least  one  device  worn  by  the  subject  within  the  environment.  In  at  least  one  alternative 
embodiment,  the  real  time  or  recorded  subject  data  are  gathered  from  environment  controls  operable  by 
the  subject  within  the  environment.  In  further  addition,  in  at  least  one  embodiment,  the  real  time  or 
recorded  data  also  include  external  world  data,  including  for  example  time  of  day,  visibility  and  noise 
level. 

[0047]  The  output  routine  104  is  operatively  associated  with  at  least  one  output  device  to  provide  the 
true  position  and  orientation  of  the  environment  and  the  perceived  subject  attitude.  Such  output  data  may 
be  stored  for  later  use  or  post  hoc  review,  and/or  made  immediately  available  to  the  subject  within  the 
environment  or  a  remote  operator  of  the  SD  system  100.  In  at  least  one  embodiment,  the  output  routine 
104  is  coupled  to  a  display  and  provides  VCR-like  controls  (e.g.,  play,  pause,  stop,  etc...),  an  attitude 
indicator  and  3D  animation  of  actual  vs.  perceived  environment  attitude. 

[0048]  The  countermeasure  routine  106  is  operable  to  provide  a  multisensory  approach  to 
preventing,  minimizing  or  compensating  for  subject  SD.  More  specifically,  the  countermeasure  routine 
106  is  operable  to  initiate  a  range  of  different  countermeasures  including  visual,  auditory,  olfactory  and 
tactile  actions.  As  the  probability  of  an  SD  event  increases,  the  countermeasure  routine  106  is  also 
operable  to  increase  the  implemented  countermeasure  from  cautionary  (e.g.,  flashing  a  warning  or 
sounding  an  alarm)  to  emergency  (e.g.,  engaging  autopilot  or  ejecting  the  pilot). 

[0049]  The  Vestibular  Attitude  Calculator  (VAC)  routine  108  is  operable  to  perform  all  calculations 
associated  with  the  vestibular  system  including  Threshold  assessment  of  acceleration  in  each  axis, 
vestibular  Washout  in  each  axis  and  perceived  attitude  deltas.  This  is  accomplished  in  at  least  one 
embodiment  through  the  Threshold  adjustment  routine  110,  the  Washout  routine  112  and  the  vestibular 
illusion  routine  114.  In  at  least  one  embodiment  the  vestibular  illusion  routine  includes  a  somatogravic 
routine  116  (predicting  illusions  caused  by  change  in  linear  accelerations  and  decelerations  or  gravity  that 
affect  the  otolith  organs),  and  a  somatogyral  routine  118  (predicting  illusions  caused  by  angular 
accelerations  and  decelerations  stimulating  the  semicircular  canals). 

[0050]  So  as  to  effectively  and  advantageously  initiate  the  most  appropriate  countermeasure  for  a 
perceived  SD  event,  SD  system  100  is  not  only  capable  of  determining  the  probability  of  an  SD  event 
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occurring,  but  in  at  least  one  embodiment  is  also  capable  of  selectively  identifying  at  least  three  types  of 
SD  illusions,  namely,  the  Leans  illusion,  the  Coriolis  illusion,  and  the  Graveyard  Spiral  illusion.  The 
probabilistic  determination  that  a  Leans,  Coriolis  or  Graveyard  Spiral  illusion  is  occurring  is  based  on 
time  sequence  modeling. 

The  Leans  Illusion 

|0051]  The  Leans  illusion  is  one  of  the  most  common  vestibular  based  disorientations,  and  is 
primarily  associated  with  the  erroneous  perception  of  changes  in  hank  angle.  The  Leans  results  from  a 
subject's  failure  to  detect  angular  roll  or  banking  motion.  During  continuous  straight  and  level  motion, 
the  subject  will  correctly  perceive  that  he  or  she  is  straight  and  level.  However,  if  a  hank  is  entered 
slowly  (below  Threshold)  or  maintained  for  a  prolonged  time,  the  fluid  in  the  semicircular  canals  of  the 
ear  will  stabilize  and  Washout  will  occur.  If  the  subject  is  quickly  returned  to  straight  and  level,  the 
motion  of  the  fluid  in  the  semicircular  canals  will  give  the  sensation  that  the  subject  is  banking  in  the 
opposite  direction,  and  the  subject  will  have  a  tendency  to  lean  back  towards  the  original  hank  orientation 
as  an  attitude  erroneously  perceived  to  be  straight  and  level. 

[0052]  VAC  routine  108  models  the  Leans  illusion  as  a  timed  sequence  of  events  with  the 
probability  of  assessment  of  the  disorientation  increasing  with  each  successive  event.  As  indicated  in 
FIG.  3,  the  first  event  300  is  the  initiation  of  a  roll  at  a  rate  below  the  vestibular  threshold.  Research 
investigating  response  to  aircraft  movements  suggests  that  in  at  least  one  embodiment  wherein  the 
environment  is  an  aircraft  and  the  subject  is  a  pilot,  the  default  vestibular  threshold  values  are  3.2°/sec  for 
roll,  2.6°/sec  for  pitch  and  l.l°/sec  for  yaw.  The  threshold  adjustment  routine  permits  these  values  to  be 
adjusted  for  individual  subjects.  As  shown  in  FIG.  3,  the  model  is  for  the  roll  axis;  however  additional 
models  exist  for  pitch  and  yaw  as  well. 

[0053]  The  second  event,  302  is  a  roll  angle  of  greater  than  5  degrees  that  lasts  longer  than  5 
seconds.  When  the  two  events  occur  in  sequence,  it  is  possible  that  the  pilot  has  not  noticed  the  ensuing 
roll  angle  and  as  such  that  there  is  a  difference  between  the  subject's  perceived  attitude  and  the  true 
attitude  of  the  environment.  As  such  the  method  indicates  a  possibility  of  an  SD  event,  but  only  at  a  very 
low  confidence  level. 

|0054]  The  third  event  304  is  the  loss  of  altitude  as  measured  by  negative  vertical  velocity.  In  the 
model  illustrated  this  is  three  hundred  feet  per  minute,  although  this  value  is  also  adjustable.  If  this  event 
follows  events  300  and  302  in  sequence,  it  is  possible  that  the  subject  has  not  noticed  the  loss  of  altitude 
and  the  method  represents  an  increased  confidence  in  the  assessment  of  the  Leans. 

[0055]  The  fourth  event  306  is  a  roll  well  above  the  vestibular  threshold  in  the  opposite  direction 
from  that  of  the  first  event.  If  this  occurs  in  sequence  following  events  300,  302  and  304,  it  is  possible 
that  the  subject  has  noticed  the  roll  angle  and  has  quickly  corrected  back  towards  level.  When  this  occurs, 
the  subject's  vestibular  system  registers  a  roll  in  the  opposite  direction,  again  resulting  in  a  difference 
between  the  perceived  attitude  and  the  actual  attitude  of  the  environment.  At  this  point  the  method 
represents  a  high  level  of  SD  event  certainty. 

[0056]  The  fifth  and  final  event  308  is  the  tilt  of  the  pilot's  head  opposite  the  perceived  roll  angle.  If 
this  occurs  following  events  300,  302,  304  and  306,  it  is  likely  that  the  pilot  is  experiencing  the  Leans 
illusion  and  the  model  represents  an  even  higher  level  of  SD  event  certainty. 

The  Coriolis  Illusion 

|0057]  The  Coriolis  illusion  is  the  most  dangerous  of  all  vestibular  illusions,  causing  overwhelming 
disorientation.  The  Coriolis  illusion  involves  the  simultaneous  stimulation  of  two  semicircular  canals.  It 
occurs  when  a  subject  experiencing  a  prolonged  turn  makes  a  sudden  head  motion  in  a  different 
geometrical  plane  from  the  plane  of  the  turn  (such  as  by  suddenly  tilting  the  head  forward  or  backwards). 
[0058]  When  in  a  prolonged  turn,  the  semicircular  canal  corresponding  to  the  yaw  axis  will  equalize. 
The  endolymph  fluid  in  the  semicircular  canals  no  longer  deviates,  or  moves  the  cupula;  thus  the  hairs  of 
the  cupula  are  not  bent.  If  the  person  initiates  a  head  movement  in  a  geometrical  plane  other  than  that  of 
the  turn,  the  yaw  axis  semicircular  canal  is  moved  from  the  plane  of  rotation  to  a  new  plane  of  non- 
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rotation.  The  fluid  then  slows  in  that  canal,  resulting  in  a  sensation  of  a  turn  in  the  opposite  direction  of 
the  original  turn.  Simultaneously,  the  other  two  canals  are  brought  within  a  plane  of  rotation,  the  fluid 
stimulation  in  those  other  two  canals  creates  the  perception  of  motion  in  three  different  planes  of  rotation 
-  yaw,  pitch  and  roll  -  all  at  the  same  time. 

[0059]  When  a  person's  head  is  suddenly  tilted  or  moved  in  one  direction  or  another,  the  brain 
usually  is  able  to  compensate  very  quickly  with  involuntary  eye  movement  in  the  opposite  direction.  It  is 
this  behavior  that  permits  a  person  to  maintain  visual  fixation  upon  an  object  while  his  or  her  body  is 
being  jostled  about.  This  typically  helpful  involuntary  behavior  can  be  a  serious  problem  when 
experiencing  the  Coriolis  illusion,  for  the  subject  may  experience  very  rapid  involuntary  eye  movement 
that  further  leads  to  feelings  of  disorientation.  In  addition,  the  involuntary  eye  motion  often  makes  it 
impossible  for  the  subject  to  visually  perceive  his  or  her  actual  orientation  either  from  the  horizon, 
instruments,  or  other  objects.  Visual  warning  systems  or  indicators  are  also  effectively  mooted  during  an 
episode  of  the  Coriolis  illusion. 

[0060]  Based  on  this  description,  an  embodiment  of  the  method  employs  a  model  of  the  Coriolis 
illusion  as  a  time  sequence  of  events  that  include  the  Washout  effect  on  the  vestibular  system  associated 
with  the  prolonged  constant-rate  turn.  While  the  Coriolis  illusion  primarily  occurs  when  Washout  occurs 
in  the  yaw  plane,  the  model  also  accounts  for  the  potential  for  Washout  in  both  pitch  and  roll. 

[0061]  More  specifically,  as  shown  in  FIG.  4,  the  first  event  400  is  an  above  threshold  acceleration 
in  any  of  the  three  planes.  The  second  event  402  requires  that  the  acceleration  from  the  first  event  400 
result  in  a  sustained  rate  of  rotation  sufficient  to  induce  Washout.  Where  the  method  is  embodied  in  the 
SD  system  100,  the  Washout  level  is  calculated  by  the  VAC  routine  108.  In  at  least  one  embodiment,  the 
level  of  Washout  achieved  in  the  second  event  402  is  at  least  50%.  As  shown  in  FIG.  4,  the  model  is  for 
the  roll  axis;  however  additional  models  exist  for  pitch  and  yaw  as  well. 

[0062]  The  third  event  404  is  a  large  movement  of  the  head  in  a  plane  other  than  the  plane  of 
acceleration  while  the  conditions  of  the  first  and  second  events  400,  402  are  maintained.  If  this  large 
motion  of  the  head  is  detected,  the  probability  of  a  Coriolis  illusion  occurring  as  an  SD  event  is  quite 
high.  Although  it  is  ideal  to  perceive  the  true  orientation  of  the  subject's  head,  such  as  when  the  subject  is 
wearing  a  motion  sensing  helmet,  the  Coriolis  illusion  can  also  be  predicted  by  observing  awkward  and/or 
radically  inappropriate  operations  of  the  subject's  controls  and/or  the  attitude  of  the  environment  itself 
following  the  onset  of  the  first  and  second  events  400,  402. 

The  Graveyard  Spiral  Illusion 

[0063]  The  Graveyard  Spiral  illusion  is  yet  another  somatogyral  illusion  commonly  associated  with 
fixed-wing  aircraft.  For  example,  the  subject  enters  a  turn  (constant  rotation)  and  remains  in  that  turn  for 
a  sufficient  time  to  induce  Washout.  This  is  then  followed  by  correcting  back  to  a  straight  and  level  path. 
The  equalized  fluid  in  the  semicircular  canals  of  the  subject  now  moves  in  the  opposite  direction  inducing 
a  strong  sensation  of  rotation  in  the  opposite  direction  to  the  initial  turn.  To  correct  for  this  illusion  the 
subject  will  then  control  the  environment  to  correct  for  the  perceived  turn,  an  action  that  actually  puts  the 
environment  back  into  the  original  rotation;  however,  the  subject  feels  that  he  or  she  is  traveling  straight. 
[0064]  Based  on  this  description,  an  embodiment  of  the  method  employs  a  model  of  the  Graveyard 
Spiral  illusion  as  a  time  sequence  of  events  of  rotations  in  the  yaw  axis.  As  shown  in  FIG.  5,  the  first 
event  500  is  sustained  rotation  sufficient  to  result  in  a  high  level  of  Washout  (e.g.,  75%)  in  the  yaw  axis. 
This  is  then  followed  by  the  second  event  502,  an  above-threshold  acceleration  in  the  opposite  direction  - 
the  standard  action  to  end  the  original  turn.  The  third  action  504  is  an  above  threshold  yaw  acceleration 
in  the  original  direction  prior  to  the  subject  recovering  from  the  Washout  effect,  an  event  indicating  a 
reasonable  probability  of  an  SD  event.  The  model  of  FIG.  5  is  for  the  yaw  axis;  however  additional 
models  exist  for  pitch  and  roll  as  well. 

[0065]  In  the  case  where  the  environment  is  an  aircraft,  the  fourth  event  506  is  negative  vertical 
velocity  while  banked  in  the  unperceived  turn.  Negative  vertical  velocity  is  the  loss  of  altitude  due  to  the 
loss  of  lift  while  in  the  unperceived  turn.  Such  an  event  raises  the  probability  of  the  SD  event  to  yet  a 
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higher  level  of  confidence.  A  fifth  event  508  is  the  pitch  up  of  the  control  inputs  while  banked,  causing 
an  increasingly  tighter  downward  spiral. 

[0066]  This  fifth  event  is  due  to  the  subject  (e.g.,  pilot)  feeling  no  rotation  but  being  aware  of  the 
loss  in  altitude  and  pulling  back  on  the  stick.  Were  he  or  she  in  the  perceived  straight  and  level  flight,  this 
action  would  result  in  the  environment  (e.g.,  aircraft)  climbing.  As  he  or  she  is  actually  in  a  banked  turn, 
pulling  back  on  the  stick  places  the  environment  (e.g.,  aircraft)  into  a  tighter  spiraling  descent.  This  fifth 
event  raises  the  probability  of  the  SD  event  yet  higher. 

[0067]  To  advantageously  predict  with  increasing  confidence  levels  of  the  onset  of  vestibular 
illusions,  in  at  least  one  embodiment,  a  state  table  is  employed.  More  specifically,  the  VAC  routine  108 
of  SD  system  100  is  associated  with  a  state  table.  The  state  table  maintains  the  state  of  the  subject,  the 
environment  and  the  external  world  (if  the  method  or  system  is  collecting  external  world  data),  and 
permits  the  vestibular  illusion  routine  (e.g.,  vestibular  illusion  routine  114)  to  track  the  progression  of 
events,  such  as  those  described  above  with  respect  to  the  Leans,  Coriolis  and  Graveyard  Spiral  illusions. 
The  state  table  may  also  be  used  to  track  the  increasing  confidence  level  that  an  SD  event  is  occurring, 
and  to  determine  whether  the  subject  has  responded  to  an  initiated  countermeasure. 

[0068]  Whereas  prior  systems,  such  as  U.S.  5,269,848  to  Repperger,  use  a  specific  threshold  of 
difference  between  actual  attitude  and  a  Kalman  filter  model  of  attitude  to  determine  whether  an  SD  event 
is  or  is  not  occurring,  the  SD  system  100  and  disclosed  method  provides  a  sliding  scale  of  confidence  in 
predicting  whether  or  not  an  SD  event  is  occurring.  Such  a  sliding  scale  of  confidence  prediction  permits 
appropriate  countermeasures  to  be  implemented  before  a  true  crisis  might  develop,  and  for  the 
implemented  countermeasures  to  increase  from  cautionary  to  emergency  as  the  probability  of  the  SD 
event  increases. 

[0069]  SD  system  100  is  advantageously  operational  either  as  an  onboard  system  for  combating 
spatial  disorientation  or  as  a  post-hoc  analysis  tool.  FIG.  6  illustrates  a  general  high  level  flow  diagram  of 
an  embodiment  of  a  method  applied  onboard  an  environment,  such  as  for  example  an  aircraft,  and  FIG.  7 
illustrates  a  general  high  level  flow  diagram  of  an  embodiment  of  a  method  applied  for  post  hoc  review. 
It  is  understood  and  appreciated  that  the  described  processes  need  not  be  performed  in  the  order  in  which 
they  are  herein  described,  but  that  this  description  is  merely  exemplary  of  at  least  one  preferred  method. 
[0070]  As  illustrated  in  the  real  time  detection  environment  of  FIG.  6,  in  at  least  one  embodiment, 
the  method  commences  with  the  real  time  collection  of  environment  data,  block  600.  The  environment 
data  include  the  true  position  and  orientation  of  the  environment. 

[0071]  Real  time  data  are  also  collected  from  the  subject  within  the  environment,  block  602.  In  at 
least  one  embodiment,  the  subject  data  include  the  true  position  and  orientation  of  the  subject,  and  more 
specifically  the  subject's  head,  within  the  environment.  Subject  modifiable  parameters  such  as  timing  and 
Washout  adjustments  are  in  at  least  one  embodiment  established  before  activity  in  the  environment  (e.g., 
flight)  commences. 

[0072]  In  at  least  one  embodiment,  optional  external  world  data  are  also  collected  as  indicated  by 
optional  block  604.  Such  external  world  data  include  time  of  day  and  visibility,  and  may  include  other 
information  such  as,  for  example,  level  of  ambient  noise  or  temperature.  For  example,  a  photo  sensor 
may  be  used  to  determine  whether  it  is  bright  and  sunny,  overcast  or  dark. 

[0073]  It  is  understood  and  appreciated  that  environment  data,  subject  data  and  external  world  data 
are  collected  at  regular  time  intervals.  Such  time  intervals  may  be  on  the  order  of  seconds  or  fractions  of 
seconds  such  that  the  data  appear  as  a  continuous  stream  of  data  elements.  Such  data  may  also  be 
collected  at  different  intervals  with  the  interval  decreasing  as  the  onset  of  events  indicates  the  developing 
possibility  of  an  SD  event. 

[0074]  With  respect  to  situations  where  the  environment  is  an  aircraft  and  the  subject  is  a  pilot,  in  at 
least  one  embodiment,  the  environment  data  are  gathered  from  one  location,  such  as  the  aircraft's  center 
of  gravity,  and  the  subject  data  are  received  from  a  different  location,  such  as  where  the  pilot  is  sitting. 
As  the  pilot  is  typically  located  some  distance  away  from  the  aircraft's  center  of  gravity,  the  effects  of 
roll,  pitch  and  yaw  upon  the  pilot  are  different  from  their  effects  upon  the  center  of  gravity  of  the  aircraft. 
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|0075]  The  perceived  attitude  of  the  subject  is  then  calculated  for  each  increment  of  time  for  which 
there  is  collected  environment  data,  block  606.  In  addition,  the  Washout  of  the  subject  is  also  predicted. 
An  evaluation  is  then  performed  collectively  upon  the  environment  data,  the  subject  data,  the  perceived 
subject  data  and  Washout  timing,  block  608. 

[0076]  It  is  to  be  understood  and  appreciated  that  Washout,  while  important  for  some  illusions,  may 
be  a  non-factor  for  others  such  as  the  Leans  illusion  where  it  is  the  attitude  changes  below  human 
threshold  levels  that  are  at  issue.  In  such  instances,  the  evaluation  of  Washout  as  effectively  zero  does  not 
alter  the  prediction,  as  the  environment  data  indicate  the  subtle  onset  of  time  sequence  events  that  will 
identify  the  SD  event. 

[0077]  More  specifically,  in  at  least  one  embodiment,  the  above  models  of  the  Leans,  Coriolis  and 
Graveyard  Spiral  illusions  are  employed.  If  the  collected  data  indicate  a  probability  for  an  SD  event 
under  any  of  these  models,  decision  610,  the  method  directs  that  countermeasure  actions  be  taken.  Of 
course,  it  is  entirely  within  reason  that  the  probability  of  an  SD  event  is  effectively  zero.  In  such  a  case, 
the  data  are  recorded  for  later  use  and  post  hoc  analysis,  block  612  (including  at  least  environmental  data 
and  subject  data).  If  the  operations  with  the  environment  are  continuing  (e.g.,  still  in  flight),  the  method 
continues  with  the  collection  of  data,  returning  to  block  600. 

[0078]  If,  as  in  decision  610,  there  is  a  probability  of  an  SD  event,  the  degree  of  probability,  i.e.,  the 
certainty  of  the  event,  is  evaluated  in  decision  614.  Where  the  probability  is  low,  a  cautionary 
countermeasure  is  implemented,  block  616.  Such  a  cautionary  countermeasure  may  include  a  warning 
light,  auditory  warning  or  other  action  selected  to  help  inform  the  subject  that  an  SD  event  may  be 
occurring.  Where  the  probability  is  high,  decision  616,  a  more  intense  countermeasure  is  initiated,  such 
as  an  emergency  countermeasure,  block  618. 

[0079]  With  respect  to  the  evaluation  of  data,  block  608,  the  collection  of  external  world  data  in  at 
least  one  embodiment  advantageously  improves  calculation  of  the  probability  of  an  SD  event.  More 
specifically,  in  a  setting  where  visibility  is  clear  and  the  subject  has  visual  awareness  of  the  geography 
around  him  or  her,  and  from  that  awareness  a  perception  of  his  or  her  attitude  and  orientation,  the 
vestibular  illusions  may  be  significantly  thwarted  by  the  subject's  brain.  The  predictive  method,  such  as 
that  embodied  by  SD  system  100  may  therefore  lower  the  prediction  of  an  SD  event  in  light  of  the 
external  world  data. 

[0080]  However,  where  visibility  is  limited  and  it  is  not  possible  to  visually  discern  the  surrounding 
geography,  such  as  an  when  flying  during  overcast  conditions  or  in  haze,  the  predictive  method,  such  as 
that  embodied  by  SD  system  100  may  raise  the  prediction  of  an  SD  event  in  light  of  the  external  world 
data. 

[0081]  By  predicting  not  only  the  probability  of  an  SD  event  but  also  the  type  of  SD  event,  the 
choice  of  countermeasure  implemented  to  combat  the  SD  event  is  advantageously  improved.  For 
example,  if  a  subject  is  experiencing  a  Coriolis  illusion,  a  blinking  light  or  auditory  alert  will  likely  be  of 
little  value  to  the  subject  or  the  environment,  whereas  engaging  autopilot  or  ejecting  the  subject  may  save 
either  or  both  of  the  pilot  and  the  aircraft.  In  addition,  warning  lights  as  countermeasures  may  have 
diminished  effectiveness  in  a  bright  setting,  just  as  auditory  alerts  may  be  diminished  in  loud  settings. 
Likewise,  in  an  increased  G  setting,  tactile  countermeasures  may  be  masked  or  overshadowed  by  the 
forces  already  at  play  upon  the  pilot's  body,  thus  making  a  strobe  light  or  audio  countermeasure  more 
effective.  The  selectivity  of  different  countermeasures,  as  well  as  the  degree  of  the  countermeasure  (e.g., 
unique  display  symbol,  blinking  light  to  flashing  strobe,  audio  warning  to  alarm  siren  to  changing  the 
pitch  of  a  continuous  artificial  wind  sound,  activation  of  a  warning  signal  to  flashing  messages  across 
multiple  displays,  tactile  sensations  of  temperature  to  blasts  of  cold  air,  tactile  sensations  of  pressure  or 
vibration  (such  as  from  a  vest  or  seat)  to  electrical  shock,  mild  to  extreme  odors,  engaging  auto-pilot  or 
auto-recovery  to  automated  ejection  of  the  subject,  recorded  verbal  requests  to  recorded  verbal  orders, 
etc...)  permits  the  system  to  administer  the  most  effective  countermeasure  or  countermeasures  so  as  to 
effectively  combat  the  SD  event. 

[0082]  In  at  least  one  embodiment,  subject  workload  data  are  also  received  from  the  subject. 
Workload,  as  in  what  the  subject  is  doing,  may  increase  or  decrease  the  subject's  susceptibility  to  SD 
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events  and  may  alter  the  effectiveness  of  certain  countermeasures.  Human  beings  are  capable  of 
performing  multiple  tasks  simultaneously;  this  ability  is  otherwise  known  as  parallel  processing. 

[0083]  In  parallel  processing,  human  beings  use  sensory  channels,  processing  resources,  and 
response  channels  (somatic,  auditory,  visual,  vestibular,  olfactory,  psychomotor-primary,  psychomotor¬ 
secondary  and  cognitive)  used  to  greater  or  lesser  degrees  for  different  tasks.  If  the  visual  channels  and 
auditory  channels  are  in  high  use,  such  as  when  approaching  for  landing  and  communicating  with  the 
control  tower,  the  subject  pilot  will  likely  respond  more  quickly  to  countermeasures  administered  to  less 
involved  channels,  such  as  the  olfactory  or  tactile.  Moreover,  in  an  embodiment  utilizing  subject 
workload  data,  countermeasures  are  further  selected  for  non-taxed  workload  channels. 

[0084]  Whether  based  on  external  world  data,  workload  data  or  combinations  thereof,  selectively 
choosing  from  a  panel  of  countermeasures  is  advantageous.  Selectively  choosing  from  a  panel  of 
countermeasures  including,  but  not  limited  to,  auditory,  visual,  olfactory,  tactile  and  mechanical 
intervention  provides  a  significant  advantage  in  providing  at  least  one  countermeasure  with  the  highest 
likelihood  of  being  acknowledged  and  acted  upon  by  the  subject  in  combating  SD. 

[0085]  Following  the  implementation  of  a  countermeasure,  the  data  (including  at  least  environmental 
data  and  subject  data)  are  recorded  for  later  use  and  post  hoc  analysis,  block  612.  In  at  least  one 
alternative  embodiment,  the  perceived  subject  attitude  is  also  recorded.  If  the  operations  with  the 
environment  are  continuing  (e.g.,  still  in  flight),  the  method  continues  with  the  collection  of  data, 
returning  to  block  600.  Such  a  return  further  aids  in  evaluating  the  effectiveness  of  the  implemented 
countermeasure,  blocks  616,  618.  If  the  collected  data  again  indicate  the  probability  of  an  SD  event, 
decision  610,  the  countermeasure  is  continued,  increased  or  augmented  with  additional  countermeasures. 
[0086]  The  analysis  capabilities  of  the  SD  system  100  provide  significant  advancement  in 
understanding  SD  events  and  devising  further  methods  and  systems  to  overcome  them.  For  example, 
flight  safety  researchers  can  utilize  the  SD  system  100  not  only  in  accident  reconstruction,  but  also  to 
identify  situations  that  might  tend  to  induce  or  increase  the  opportunity  for  SD  events  to  occur.  FIG.  7 
illustrates  a  high  level  embodiment  of  this  post  hoc  review. 

[0087]  In  order  to  act  in  a  post  hoc  manner,  environment  data  and  subject  data  from  a  subject  within 
the  environment  must  be  pre-recorded.  As  indicated  in  blocks  700,  702,  these  data  are  collected  by  the 
SD  system  100.  Moreover,  the  stream  of  data  may  be  read  into  a  memory  array,  accessed  sequentially  or 
otherwise  made  available  to  the  input  routine  102  of  SD  system  100  as  is  appropriate  for  the  physical 
embodiment  of  SD  system  100  and  the  volume  of  data  provided.  In  at  least  one  embodiment,  the  subject 
Threshold  values  and  Washout  values,  as  well  as  the  illusion  model  parameters,  are  also  collected  so  as  to 
enhance  tailoring  the  post  hoc  review  to  a  particular  subject  and  or  set  of  conditions,  as  indicated  by 
optional  block  704. 

[0088]  To  provide  a  sequence  to  the  events,  in  at  least  one  embodiment,  the  environment  data  are 
received  as  a  series  of  distinct  data  elements,  such  as  time-stamped  data  packets.  Commencing  with  the 
first  environment  data  element,  the  perceived  subject  attitude  is  calculated  for  the  environment  data 
element,  block  706. 

[0089]  In  accordance  with  at  least  the  above  models  (Leans,  Coriolis  and  Graveyard  Spiral  illusion), 
the  environment  data,  subject  data  and  perceived  subject  attitude  are  evaluated  to  determine  the 
probability  of  an  SD  event,  block  708.  The  probability  as  calculated  is  then  reported  to  the  SD  system 
user  performing  the  post  hoc  review,  block  710.  In  at  least  one  embodiment,  such  a  report  is  made  via  a 
display  screen  upon  which  the  environment  data,  subject  data,  perceived  subject  attitude  and  probability 
of  an  SD  event  are  graphically  represented. 

[0090]  If  the  end  of  the  environment  data  has  been  reached,  decision  712,  the  post  hoc  analysis 
process  ends.  If  more  elements  of  environment  data  remain,  the  process  increments  to  the  next  element  of 
environment  data,  block  714,  and  re-calculates  the  perceived  subject  attitude,  block  706. 

[0091]  In  at  least  one  embodiment,  the  post  hoc  review  process  further  includes  a  review  of  which 
countermeasures  were  initiated  when  the  SD  event  was  evaluated  in  real  time,  thus  permitting  the  post 
hoc  reviewer  to  evaluate  the  effectiveness  of  the  countermeasures. 
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[0092]  This  post  hoc  review  advantageously  permits  a  researcher  to  identify  similarities  and 
differences  between  the  collected  data  and  similar  pre-recorded  event  data.  For  example,  flight  patterns 
over  certain  geographic  regions,  or  occurring  at  certain  times  of  day  or  in  certain  types  of  weather  may  be 
identified  as  posing  a  greater  risk  of  SD  events  to  pilots.  In  at  least  one  embodiment,  this  comparison 
review  process  may  be  incorporated  as  an  automated  component  of  the  SD  system  100.  In  other  words, 
when  evaluating  environment  data,  subject  data  and  perceived  attitude  data  in  real  time,  an  enhanced  SD 
system  100  may  also  review  similar  data  from  pre-recorded  events  to  further  enhance  the  prediction  of  an 
SD  event. 

[0093]  Changes  may  be  made  in  the  above  methods,  systems  and  structures  without  departing  from 
the  scope  hereof.  It  should  thus  be  noted  that  the  matter  contained  in  the  above  description  and/or  shown 
in  the  accompanying  drawings  should  be  interpreted  as  illustrative  and  not  in  a  limiting  sense.  The 
following  claims  are  intended  to  cover  all  generic  and  specific  features  described  herein,  as  well  as  all 
statements  of  the  scope  of  the  present  method,  system  and  structure,  which,  as  a  matter  of  language,  might 
be  said  to  fall  therebetween. 


CLAIMS 


WHAT  IS  CLAIMED  IS: 

1 .  A  computer-readable  medium  on  which  is  stored  a  computer  program  for  detecting,  analyzing 

and  responding  to  a  spatial  disorientation  event,  the  computer  program  comprising: 

an  input  routine  operatively  associated  with  an  input  device  for  receiving  real  time  data, 

recorded  data,  subject  preference  information  or  combinations  thereof,  the  data  including: 
environment  data  from  an  environment,  the  environment  data  including  true  position  and 
orientation  of  the  environment;  and 
subject  data  from  a  subject  within  the  environment; 
a  vestibular  attitude  calculator  routine  for  computing  the  perceived  subject  attitude  of  the 
subject  within  the  environment  based  on  the  environment  data  and  subject  data,  the 
vestibular  attitude  calculator  including: 
a  Washout  routine  to  calculate  a  Washout  value; 

a  vestibular  illusion  routine  to  calculate  a  probability  of  a  vestibular  illusion;  and 
a  Threshold  adjustment  routine  permitting  adjustment  of  Washout  thresholds  and 
vestibular  thresholds  based  on  provided  subject  preference  information; 
a  countermeasure  routine  operating  in  response  to  the  Washout  value  and  the  probability  of  the 
vestibular  illusion;  and 

an  output  routine  operatively  associated  with  an  output  device  to  provide  the  true  position  and 
orientation  of  the  environment  and  the  perceived  subject  attitude. 


2.  The  computer-readable  medium  of  claim  1,  the  Washout  value  being  calculated  as  a  non-linear 
element. 

3.  The  computer-readable  medium  of  claim  1,  the  input  routine  further  including  external  world 
data. 

4.  The  computer-readable  medium  of  claim  1,  wherein  the  countermeasure  routine  is  operatively 
associated  with  a  plurality  of  countermeasure  devices,  the  choice  and  activation  of  a 
countermeasure  determined  by  the  Washout  value,  the  probability  of  the  vestibular  illusion,  and 
combinations  thereof. 

5.  The  computer-readable  medium  of  claim  4,  wherein  the  choice  and  activation  of  the 
countermeasure  is  further  determined  by  the  environment  data. 

6.  The  computer-readable  medium  of  claim  1,  wherein  the  computer  program  is  executed  to  perform 
a  post-hoc  analysis  of  the  subject  within  the  environment. 

7.  The  computer-readable  medium  of  claim  1,  wherein  the  environment  is  an  aircraft  in  flight  and 
the  subject  is  a  pilot. 
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8.  The  computer-readable  medium  of  claim  1,  wherein  the  vestibular  attitude  calculator  further 
includes  a  workload  calculator  routine  for  calculating  the  workload  of  the  subject  within  the 
environment. 

9.  The  computer-readable  medium  of  claim  1,  wherein  the  subject  data  are  recorded  from  at  least 
one  device  worn  by  the  subject. 

10.  The  computer-readable  medium  of  claim  1,  wherein  the  subject  data  are  recorded  from  one  or 
more  environment  controls  operable  by  the  subject. 

11.  The  computer-readable  medium  of  claim  1,  wherein  the  vestibular  attitude  calculator  routine 
identifies  the  spatial  disorientation  event  as  either  a  somatogravic  illusion  or  a  somatogyral 
illusion. 

12.  The  computer-readable  medium  of  claim  11,  wherein  the  somatogyral  illusion  is  further  identified 
as  a  Leans  illusion,  a  Coriolis  illusion  or  a  Graveyard  Spiral  illusion. 

13.  The  computer-readable  medium  of  claim  11,  wherein  the  countermeasure  routine  further  operates 
in  response  to  the  identified  spatial  disorientation  event. 

14.  A  method  for  analyzing  a  spatial  disorientation  event  post-hoc,  comprising: 

collecting  environment  data  elements  recorded  from  an  environment; 
collecting  subject  data  recorded  from  a  subject  within  the  environment; 
calculating  perceived  subject  attitude  of  the  subject  within  the  environment  for  each 
environment  data  element; 

evaluating  the  environment  data,  the  subject  data  and  the  perceived  subject  attitude  to 
determine  a  probability  of  a  spatial  disorientation  event;  and 
reporting  the  probability  of  the  spatial  disorientation  event. 

15.  The  method  of  claim  14,  wherein  the  subject  data  are  collected  from  at  least  one  device  worn  by 
the  subject. 

16.  The  method  of  claim  14,  wherein  the  subject  data  are  collected  from  at  least  one  environment 
control  operable  by  the  subject. 

17.  The  method  of  claim  14,  wherein  the  spatial  disorientation  event  is  identified  as  a  vestibular 
illusion. 

18.  The  method  of  claim  17,  wherein  the  vestibular  illusion  is  further  identified  as,  a  Leans  illusion,  a 
Coriolis  illusion,  or  a  Graveyard  Spiral  illusion. 

19.  The  method  of  claim  14,  wherein  the  environmental  data  are  recorded  at  predetermined  time 
intervals,  the  perceived  subject  attitude  calculated  for  each  time  interval. 

20.  The  method  of  claim  14,  wherein  the  environment  is  an  aircraft  in  flight  and  the  subject  is  a  pilot. 

21.  The  method  of  claim  14,  wherein  post-hoc  analysis  includes  determining  any  similarities  or 
differences  between  the  collected  data  and  pre-recorded  event  data. 

22.  The  method  of  claim  14,  wherein  the  method  is  stored  on  a  computer-readable  medium  as  a 
computer  program,  which  when  executed  by  a  computer  will  perform  the  steps  of  post-hoc  spatial 
disorientation  analysis. 

23.  A  method  for  combating  spatial  disorientation,  comprising: 

collecting  real  time  environment  data  from  an  environment,  the  environment  data  including 
true  position  and  orientation  of  the  environment; 
collecting  real  time  subject  data  from  a  subject  within  the  environment; 
calculating  perceived  subject  attitude  of  the  subject  within  the  environment  for  one  or  more 
environment  data  elements  and  predicting  Washout; 
evaluating  the  environment  data,  the  subject  data,  the  perceived  subject  attitude  and  Washout 
to  determine  the  probability  of  a  spatial  disorientation  event  and  a  type  of  spatial 
disorientation  event; 

implementing,  in  response  to  the  probability  of  a  spatial  disorientation  event,  at  least  one 
countermeasure,  the  countermeasure  selectively  chosen  from  a  group  of  multi  sensory 
countermeasures  and  countermeasure  actions  based  on  the  environment  data  and  spatial 
disorientation  probability;  and 
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recording  the  environmental  data  and  subject  data  as  event  data  for  post-hoc  review. 

24.  The  method  of  claim  23,  wherein  the  Washout  is  evaluated  as  a  non-linear  element. 

25.  The  method  of  claim  23,  further  including  collecting  external  world  data,  wherein  evaluating 
includes  evaluating  the  external  world  data  to  determine  the  probability  of  a  spatial  disorientation 
event. 

26.  The  method  of  claim  23,  wherein  determining  the  probability  of  the  spatial  disorientation  event 
includes  identifying  the  spatial  disorientation  event  as  a  vestibular  illusion. 

27.  The  method  of  claim  26,  wherein  the  vestibular  illusion  is  further  identified  as  a  Leans  illusion,  a 
Coriolis  illusion,  or  a  Graveyard  Spiral  illusion. 

28.  The  method  of  claim  23,  wherein  the  environment  data  are  received  from  a  first  location  within 
the  environment  and  the  subject  data  are  received  from  a  second  location  within  the  environment, 
the  second  location  being  different  from  the  first  location. 

29.  The  method  of  claim  23,  wherein  the  group  of  countermeasures  includes  auditory,  visual, 
olfactory,  tactile,  auto-recovery,  auto-ejection  and  combinations  thereof. 

30.  The  method  of  claim  23,  further  including  receiving  subject  workload  data  from  the  subject 
within  the  environment,  wherein  implementing  comprises  implementing  a  countermeasure 
selected  for  a  non-taxed  workload  channel. 

3 1 .  The  method  of  claim  23,  wherein  as  the  probability  of  a  spatial  disorientation  event  increases,  the 
implemented  countermeasure  increases  from  a  cautionary  to  an  emergency  countermeasure. 

32.  The  method  of  claim  23,  wherein  the  event  data  are  stored  with  similar  pre-recorded  event  data, 
post  hoc  review  including  determining  similarities  or  differences  between  the  event  data  and  the 
pre-recorded  event  data. 

33.  The  method  of  claim  23,  wherein  the  method  is  stored  on  a  computer-readable  medium  as  a 
computer  program,  which  when  executed  by  a  computer  will  perform  the  steps  of  real  time  and 
post-hoc  spatial  disorientation  analysis. 

34.  A  method  for  combating  spatial  disorientation,  comprising: 

collecting  real  time  environment  data  from  an  environment,  the  environment  data  including 
true  position  and  orientation  of  the  environment; 
collecting  real  time  subject  data  from  a  subject  within  the  environment; 
comparing  the  collected  data  with  pre-recorded  event  data  to  determine  any  similarities  or 
differences  between  the  event  data  and  the  pre-recorded  event  data  and  to  predict  subject 
response  under  similar  environmental  conditions; 
calculating  perceived  subject  attitude  of  the  subject  within  the  environment  for  one  or  more 
environment  data  elements  and  predicting  Washout,  the  calculations  including  the 
predicted  subject  response; 

evaluating  the  environment  data,  the  subject  data,  the  perceived  subject  attitude  and  Washout 
to  determine  a  probability  of  a  spatial  disorientation  event  and  a  type  of  spatial 
disorientation  event; 

implementing,  in  response  to  the  probability  of  a  spatial  disorientation  event,  at  least  one 
countermeasure,  the  countermeasure  selectively  chosen  from  a  group  of  multi  sensory 
countermeasures  and  countermeasure  actions  based  on  the  environment  data  and  the 
spatial  disorientation  probability;  and 

recording  the  environmental  data  and  subject  data  as  event  data  for  post-hoc  review. 

35.  The  method  of  claim  34,  wherein  the  Washout  is  evaluated  as  a  non-linear  element. 

36.  The  method  of  claim  34,  wherein  determining  the  probability  of  the  spatial  disorientation  event 
includes  identifying  the  spatial  disorientation  event  as  a  vestibular  illusion. 

37.  The  spatial  method  of  claim  36,  wherein  the  vestibular  illusion  is  further  identified  as  a  Leans 
illusion,  a  Coriolis  illusion,  or  a  Graveyard  Spiral  illusion. 

38.  The  method  of  claim  34,  further  including  receiving  subject  workload  data  from  the  subject 
within  the  environment,  the  implemented  countermeasure  further  selected  for  a  non-taxed 
workload  channel. 
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ABSTRACT 

Provided  is  a  spatial  disorientation  (SD)  system  and  method  for  detecting,  analyzing  and  responding  to  an 
SD  event.  An  input  routine  includes  receiving  environment  data  and/or  data  from  a  subject  within  the 
environment,  e.g.,  true  position  and  orientation  of  the  environment  and/or  subject.  A  vestibular  attitude 
calculator  computes  perceived  subject  attitude  for  environment  data  elements,  and  may  compute  a 
Washout  value.  A  vestibular  illusion  routine  calculates  the  probability  of  a  vestibular  illusion.  A 
threshold  adjustment  routine  adjusts  Washout  and  vestibular  thresholds  based  on  subject  preference  data. 
Probability  and  type  of  an  SD  event  is  determined  by  evaluating  the  received  and  computed  data.  Sensory 
countermeasures  may  be  implemented  responsive  to  the  probability  of  an  SD  event.  An  output  routine 
provides  true  position  and  orientation  of  the  environment  and  perceived  subject  attitude  via  an  output 
device;  such  information  may  be  recorded  for  post-hoc  review,  a  method  of  which  is  also  provided. 
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Appendix  D.  Glossary 

2D  two-dimensional 

3D  three-dimensional 

AC  aircraft 

AFB  air  force  base 

AFRL  Air  Force  Research  Laboratory 

AFSA  Air  Force  Safety  Agency 

AFSC  Air  Force  Safety  Center 

ASA  Aviation,  Supplies,  and  Academics 

AsMA  Aerospace  Medical  Association 

ATC  air  traffic  control(ler) 

CG  center  of  gravity 

Class  A  mishap  that  costs  >  $1M,  or  has  loss  of  life,  or  causes  permanent  total  disability, 
or  destroys  an  aircraft 

Class  B  mishap  that  costs  >  $200K  and  <  $  1M,  or  permanent  partial  disability,  or 
hospitalization  of  3  or  more  people 

Class  C  mishap  that  costs  >  $20K  and  <  $200K,  or  injury  that  results  in  lost  work  after  the 
day  or  shift  it  occurred,  or  disability  (source:  http://www2.faa.gov/arp/ 
environmental/5054a/M  OAFINALVERSION.doc) 
deg  degree(s) 

F-15  USAF  two-seat  high  performance  fighter  attack  jet 

F-16  USAF  multi-role  fighter  jet 

F-22  U  S AF  ’  s  newest  stealthy  fighter  attack  j  et 

FAA  Federal  Aviation  Administration 

F-PASS  Flight-Performance  Assessment  Standardized  System  (NTI’s  desk-top  simulator 
and  data  collection  tool) 

FY  fiscal  year 

G,  Gx,  Gz  force  due  to  gravity  in  particular  axis,  if  specified 

GAT-II  General  Aviation  Trainer,  second  generation  -  SD  demonstration  simulator 

HDD  head-down  display 

HECI  Crew  Systems  Development  Branch  (part  of  HEC,  the  Crew  System  Interface 

Division,  which  is  part  of  the  Human  Effectiveness  Directorate,  HE) 

HUD  head-up  display 

Hz  Hertz  (cycles  per  second;  so,  2  Hz  is  2  cycles  per  second) 

ICD  interface  control  document 

IFR  instrument  flight  rules 

IMC  instrument  meteorological  conditions 

IRB  institutional  review  board 

MA&D  Micro  Analysis  &  Design 

MAAD  Micro  Analysis  &  Design 

mag  magnetic 

min  minute(s) 

MOD  Ministry  of  Defence  (UK) 

MRT  (Wickens’)  multiple  resource  theory 

N/A  not  applicable 
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NAMRL 

NASA 

NTSB 

ONR 

PCT 

Perceived 

PPA 

R&D 

SAVVOY 

SBIR 

SD 

SDAT 

sec 

SIB 

SO 

SOAS 

SoW 

Type  1  SD 
Type  2  SD 
Type  3  SD 
UAV 
UK 

us,u.s. 

USAF 

USAFA 

USD 

VAC 

VACP 

VFR 

VMC 


Naval  Aeromedical  Research  Laboratory 
National  Aeronautics  and  Space  Administration 
National  Transportation  Safety  Board 
Office  of  Naval  Research 
Patent  Cooperation  Treaty 

Vestibular  Attitude  Calculator  value  for  an  attitude  angle  or  motion 
provisional  patent  application 
research  and  development 

somatic,  auditory,  visual,  vestibular,  olfactory,  psychomotor  and  cognitive 
workload  scores 

Small  Business  Innovation  Research 
spatial  disorientation 
Spatial  Disorientation  Analysis  Tool 
second(s) 

Safety  Investigation  Board 
spatial  orientation 
Spatial  Orientation  Aiding  System 
(contractual)  Statement  of  Work 
unrecognized  SD 
recognized  SD 
incapacitating  SD 
unmanned  aerial  vehicle 
United  Kingdom  (England) 

United  States 
United  States  Air  Force 
USAF  Academy 
unrecognized  SD 
Vestibular  Attitude  Calculator 

visual,  auditory,  cognitive,  psychomotor  -  resource  components  which  comprise 

Wickens’  multiple  resource  theory 

visual  flight  rules 

visual  meteorological  conditions 
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